How valuable are AI readiness assessments? Part 2

Previously on Keeping up with AI assessment tools…

In case you missed it, IT Brew has been trying to determine the true effectiveness of AI assessments, the tools that companies use to gauge their readiness for AI projects. According to technology firm Virtasant, more than one-third (37%) of companies underestimate the importance of these tools.

In our last article, we tested Avanade. For each assessment, we’ve taken the role of a fictional organization early in its AI journey that would likely need a lot of internal work to pursue large AI projects. The ultimate goal of our testing isn’t to compare end results, but to evaluate the types of insight and recommendations offered by each exam.

Got that? Great! Now let’s explore assessments offered by the Data Warehousing Institute (TDWI) and GrayCyan.

TDWI. First on the list is TDWI’s AI Readiness Assessment. I caught up with Fern Halper, VP of research and senior director of TDWI research for advanced analytics, who put the tool’s framework together.

She said TDWI has released a number of assessments for different areas, including assessing an organization’s readiness for agentic AI, and that the tool should serve as a starting point for companies exploring AI.

“This is sort of a first pass of ‘Here’s where you are,’ and then…based on where you are, you want to know where you’re going,” Halper said.

The tool offers upward of 70 questions in the following areas: organizational, data, skills, operational, and governance readiness. Unlike Avanade’s assessment, the questions covered a lot of territory and there was less need to debate between two answer choices as frequently. At the end, we were not only presented with my results across each evaluation area, but also greeted with a guidebook (delivered via email) that helped make sense of it.

Our fictitious organization received an overall score of 43.05 out of a maximum of 100. Based on the supplementary guide, we learned that our company was in a “strategizing” stage for most of the framework, meaning that while its leadership team may have started an AI project, there was a need for a “master plan or strategy to bring it coherence.”

Recommendations on how to proceed were brief: organizations at this stage may, “at best,” implement disconnected AI systems for multiple use cases.

TDWI’s AI assessment reminded me of the educational archetype of AI readiness tools that professor at the UK’s University of Exeter Alan Brown had described; it felt like they had put a lot of thought into its framework.

GrayCyan. The last assessment on our list was GrayCyan, which launched its AI readiness scorecard tool earlier this month. Nishkam Batta, founder and CEO of GrayCyan, an AI consulting firm for manufacturing, healthcare, and B2B businesses, told IT Brew that its tool focuses on assessing the “operational frictions” within a business. GrayCyan created a model, with three indices, to specifically address this this area, which it felt was overlooked by many companies

“We created this model, which basically tells you, operationally, where you are and are you ready to implement AI,” Batta said. “And this is different from how everybody’s focusing on data.”

GrayCyan’s scorecard requires around the same time it takes to finish an episode of The Simpsons. The test tackles six business areas: administration, finance, sales, warehouse, engineering, and ERP and governance. As each section was completed, we received a visualization of where my organization stood compared to others in the industry.

The results were much more detailed than those from TDWI and Avanade. It revealed our fictional organization was at a foundational level of AI readiness and was best suited for “task-level admin automation.” It predicted there was a $431,250 annual savings opportunity from doing so—an estimate that could be extremely valuable to an IT professional.

My scorecard generated a whopping 49 recommendations based on my results, including suggestions like synchronizing my CRM, ERP, and finance platforms, and standardizing process documentation within my organization. The guidance was so thorough that I felt like I’d be able to steer my imaginary workforce toward advanced AI implementation. Ironically, GrayCyan’s tool aligned with the third type of tool on Brown’s list of different AI assessments: a benchmarking tool.

“There is a little bit of shock and awe,” Brown said. “They want you to realize you’re a one out of five and wake you up a bit.”

After seeing the colorful visuals showing how far behind our fake company was compared to the rest of the industry, it was indeed awakening.

What this means for you. After taking these assessments from three different companies, it was clear there are a range of resources available to IT pros on their AI journeys, whether a short questionnaire or an in-depth evaluation.

Still, participants need a level of honesty to make the most out of these tools. Reich agreed that integrity is a necessary part of these processes.

“It’s like garbage in, garbage out in some aspects of it,” Reich said. “So it’s like, are you going to be transparent enough?”

Halper added that professionals may find it beneficial to take multiple exams at once for more refined results: “You could see how you stack up across a different number of them.”

Finally, Brown said the exams shouldn’t be a one-and-done deal, but rather a tool that can be used repeatedly as an organization moves through its AI implementation. Individuals should clarify what they want out of the tool in the context of their grander strategy: “It’s being clear up-front [about] what you want out of it, and matching the assessment to your need,” he said.