Amazon Stores CISO: AI pilots are ‘one of the best gifts you can have’

If at first you don’t succeed, dust yourself off and try again—advice that also applies to AI pilots that don’t go as planned.

According to a July 2025 MIT study, 95% of generative AI pilots produce zero return on investment (ROI). Amazon Stores CISO Hudson Thrift told IT Brew in an interview at RSAC that failures are an expected part of experimenting with AI.

“They’re not only inevitable, they’re one of the best gifts you can have,” Thrift told IT Brew. “You often learn more from failures that you have than the successes you have.”

Pick your poison. Thrift sat down with IT Brew to discuss how companies can make the best of failed AI pilots, emphasizing the importance of having good purpose and intention behind these initiatives. He said Amazon Stores, for example, typically only pursues AI projects where the technology “makes sense” and seems capable of living up to the claims.

“Automated penetration testing is a great example,” Thrift said. “I think a lot of us look at that and say, ‘Yep, we think that the models are capable enough today to be doing this.’”

Thrift added that Amazon Stores typically aims for a 10x ROI from its AI pilots. Having such a clear goal in mind allows it to choose carefully what “bets” it will make.

“We’ll choose to stop funding a lot of that stuff at that point and just say, ‘Let’s go on and try and find the thing that is the 10x,’” Thrift said.

Thrift said it’s important for companies to have a good understanding of what they are trying to automate when pursuing pilots and make sure the technology being applied is “appropriate” for that use case.

“If you drive a Bugatti to the grocery store, you can’t be mad that it didn’t work,” Thrift said. “It’s a Bugatti, you weren’t supposed to take it to the grocery store…Drive a sensible car that has space to put the groceries in.”

Blessings in disguise. Thrift said failures don’t always equal a bad outcome. He recalled one Amazon Stores pilot designed to make it easier for non-technical employees to work with an internal graph database by using AI to write graph queries with natural language.

“Turns out it works great…We moved a lot of users over to just talking with it, and we get questions back and it optimizes graph queries amazingly,” Thrift said. What the team hadn’t expected, though, was that separate endpoints created to allow agents to write their own graph queries eventually ended up being a preferred method for teams using the database.

“They had their agent just talk to it, just ask a question,” Thrift said. “And it turns out, in quite a few of our agentic pilots, that was a more efficient way to go about it.”

“That was super surprising for us. We thought we had built that thing for the humans to talk to the service, not for the agents to talk to the service. And pretty consistently, now we’re finding the agents prefer that path,” Thrift added.

The gift that keeps on giving. Thrift warned of common pitfalls organizations may encounter with pilot failures, including using those unsuccessful projects to downplay the benefits of AI.

“They just use it as the stalking horse for ‘AI is not working,’” Thrift said. “That’s a really scary pit to avoid because the point of failing is to learn the lesson, to try again, to succeed.”

He added companies shouldn’t forget to revisit failed pilots because AI is constantly evolving, and newer models may yield different results.

“I’ve seen a lot of engineers and business people be reticent to do that. They’re like, ‘We tried automating that last year, and it didn’t work for us,’” Thrift said. “Yeah, you guys gotta try it again. We got better.”