This AI safety org is tracking how each AI model ‘cheats’ on tasks

As AI agents get smarter, they’re also getting wilier.

A series of recent security incidents involving “rogue” AI agents have shown how they can cheat, hack, or otherwise misbehave in pursuit of their evaluation objectives. With these types of breaches in mind, the nonprofit Center for AI Safety (CAIS) developed a new benchmark, CheatBench, that aims to quantify the ways and frequency with which each of the major AI models tends to “cheat.”

Mantas Mazeika, a research scientist at CAIS and author on CheatBench, told us these incidents and mainstream discussion of catastrophic fears around the rise of AI have changed how he thinks about his work.

“It’s certainly made things feel more urgent,” Mazeika said. “It helps to focus—or it helps to narrow one’s focus—on what’s important, and maybe defer longer-term scientific understanding of the issues, and really try to get practical measurements and ways to address these problems in a shorter time frame.”

Agents have upskilled: When we last spoke with Mazeika a year ago, the highest performing agent could successfully complete freelance tasks on sites like Upwork or Fiverr a dismal 2.5% of the time. “Basically, everything [they produced] was complete garbage,” he told us earlier this month.

Now, top-performing models from OpenAI and Anthropic can manage passable work for the same tasks around a fifth of the time, per the CAIS remote labor index. Agents are not fully capable, but their trajectory is on a steep upward slope.

“It’s a crazy amount of progress if you actually look at the deliverables that these AI agents are producing for real work tasks or projects that you would find on [freelancing sites],” Mazeika said.

Are they cheating? It might be a natural question if a human student were to seemingly become eight times smarter in a year. Apparently, it’s also a pertinent one when it comes to agents.

CheatBench tracks exactly how often models take various shortcuts in service of their objectives. Many of AI’s reasoning gains have come through reinforcement learning with verifiable rewards (RLVR), or “rewarding” actions that further a quantifiable outcome. But that can also lead to reward hacking: tactics like finding a hidden answer key, copying a competitor’s work, or otherwise finding loopholes.

It was something that Mazeika said his team first noticed with the Remote Labor Index. An agent might use image generation to fake a 3D rendering, for instance. There are many ways that models can take shortcuts or “cheat” that might not rise to the level of something like a Hugging Face incident.

But another issue is that models might be inadvertently trained to cover their tracks from monitors in reinforcement learning environments.

“They are aware that they’re being monitored, and that the most efficient way for them to achieve their reward is to evade those monitors,” Mazeika said. “This is a problem that’s been discussed for a while on AI safety, and it was mainly theoretical until recently.”

What does this mean for enterprise? Do businesses thinking about model selection need to consider that Grok (78% cheat probability, per the CheatBench index), Gemini (75%), and DeepSeek (75%) all have a much greater tendency to cheat than their rivals? What should IT people make of the fact that Anthropic’s Claude Opus 5.5 is the most trustworthy, with an 11% cheat probability, while the next most honest, Muse Spark 1.3, jumps to 39%?

Mazeika said their cheating tendencies can sometimes be managed effectively with guardrails like, “Do your own work” or “Don’t look up answers online.”

“One thing that we’ve actually managed to do fairly well is to make the models extremely rabid instruction followers, almost to a fault sometimes,” he said.

But that can come with unintended consequences, too, if the directives are not worded precisely. It’s also possible to monitor systems and then address “cheating” behavior as it arises, he said.

“As the AIs get continually smarter, there might just be new ways that they can work around these restrictions in unintended manners and kind of surprise us with creativity that we wouldn’t necessarily welcome,” Mazeika said.

Know AI-self: Going forward, Mazeika would like to see more research into measuring and understanding the goals and values that are instilled in AI systems during training, even if we still aren’t able to see clearly into how these systems operate.

“This is a pretty glaring omission in our suite of evaluations for AIs. It’s hard to really cleanly focus in on the goals and values of these systems because they’re big, they’re black boxes, just huge stacks of neural connections,” Mazeika said. “And it’s clear that behaviorally they’re pursuing things that they perceive as helping them in the long run achieve some objective, but reading their minds, so to speak, is still something that’s pretty challenging to do cleanly, and something that people are working on a lot.”