How valuable are AI readiness assessments?

Somewhere out there, an IT professional is breaking into a cold sweat as they take an AI readiness assessment, anxious to see whether their organization can actually integrate the technology.

AI benchmarking assessments are all the rage these days. Large vendors like Cisco and Microsoft have released their own tools to help companies figure out their AI maturity level. And there may be good reason behind the hype: an MIT study found that 95% of generative AI pilots created zero return on investment (ROI). A company needs to know whether an expensive technology will truly pay off.

With all that in mind, we had one burning question: Are these tools as valuable as some people think? We dusted off our handy-dandy test-taking socks to find out.

Methodology. We chose free assessments from three different companies: Avanade, GrayCyan, and the Data Warehousing Institute (TDWI). With each assessment, we took on the role of an organization early in its AI journey that would likely need a lot of internal work before pursuing AI projects. The purpose of this was not to compare end results, but rather to evaluate each exam’s insights and recommendations. (In part one of this series, we’re covering Avanade; in part two, we’ll discuss GrayCyan and TDWI.)

Avanade. Avanade’s AI readiness assessment tool was first on my list. According to the company’s website, its AI readiness assessment framework categorizes users into five stages of AI readiness: exploring, planning, scaling, implementing, and realizing.

Avanade CTO and Innovation Officer Aaron Reich told IT Brew that the tool had been around for about three years and was based on two years of survey data. The goal, he added, was to provide organizations with a resource to assess their comfort level with adopting AI.

“That’s really why we ended up kind of developing the tool was to kind of give a little of that barometer to then go, ‘Oh, I hadn’t thought about something in that way,’” he said.

The test had only five high-level questions on a business’s interest in AI. After I answered those questions, the assessment deemed our faux organization as ready for the “planning” stage of AI readiness.

“You have a strategy for Al, but require help to prioritize and phase practical actions to rapidly experiment and learn,” the results read, along with a prompt guiding me to a complimentary workshop.

While that was an appropriate assessment for my fake organization, I couldn’t help but wonder: Can five questions really assess the true scope of one’s business? Would an IT pro want to sit through a lengthy assessment if they’re just trying to scope out their organization’s general readiness for AI implementation?

Alan Brown, professor in digital economy at the University of Exeter, UK, told IT Brew that there are three flavors of AI readiness tools. One serves as a funnel for additional consulting services. The other two are educational and benchmarking tools. Avanade’s tool felt like it belonged in the sales-funnel bucket.

“The idea is, you answer 10 questions, and the answer is, ‘You’re good at these three things. You’re not so good at this thing, by the way. You need one of our people and they can help you,’” Brown said, adding that these types of tests have “limited depth” because there’s a motive, such as wanting the user to chat with a sales rep or download a report.

It did feel like a CIO taking the Avanade test would probably add the workshop to their calendar to make better sense of their results and what to do next. Reich confirmed that this was the intention: “We want to be able to go, ‘Look, we have a bunch of expertise of the things that we are doing and understanding…Those learnings that we have, we’re trying to provide that to others to be able to utilize in some way.”

’Til next time. Reich said the results of AI assessment tools only provide a “snapshot in time” of an organization’s readiness. Due to the rapid pace of AI innovation, it’s also hard for these tools to remain up-to-date.

Avanade gave us a first taste of what AI tools test tools evaluate. In our next article, we test out GrayCyan’s and TDWI’s respective AI assessments to determine if AI readiness tools can actually serve as a useful tool.