How to catch and kill a rogue agent

Sponsored by

Your AI strategy has a security problem: 43% of IT leaders now cite security concerns as the top barrier to expanding AI agent deployment, according to research from JumpCloud. Before giving agents more access to apps, data, and workflows, make sure you can govern the identities behind them. See how to secure every identity, human or not.

Before a business like a consultancy or consumer mortgage lender rolls out a new agentic system, they might turn to Scale AI to try to break it.

The startup, which has tried to remake itself after a Meta quasi-acquihire last year, keeps a stable of human red teamers who simulate the intentional or unwitting antagonists an AI agent might face in the wild—casual users, trolls, hackers, or misaligned agents.

“Are [users] encouraging an agent, and is the agent simulating access it shouldn’t have? Is it claiming that it’s a human when it’s not? Is it granting a loan on terms that would be illegal in an enterprise context?” Patrick Oathout, Scale’s red team and safety lead, told us of scenarios they simulate.

Exercises like these are even more important in the agentic era, when companies may not start with a comprehensive picture of all the ways networks of autonomous bots can go awry, according to Oathout. Case in point is a headline-grabbing string of incidents in which sophisticated AI agents went rogue, slipping out of test environments and conducting complex attacks, which caught even their leading-edge makers off guard.

A recent survey from observability vendor WanAware found that nine in 10 IT leaders said they could root out rogue agents, but only 26% thought they could measure their impact within minutes. Meanwhile, a CrowdStrike report said the amount of risky behavior detected from agents is growing at 2.5 times the rate of that from humans.

As researchers quit top AI companies with warnings of grave risk, and CEOs propose a slowdown in frontier research, IT leaders say it’s time to rethink how companies oversee their growing networks of agents.

“The choice now is either [frontier companies] say, well, we can’t trace reasoning; we can’t reach full alignment; models are starting to build themselves; they’re becoming smarter than humans; we’ve got to pause,” Ariel Assaraf, CEO and co-founder of observability platform Coralogix, said. “Or we’ve got to rethink the entire framework of how we create prevention and alerting.”

Beyond guardrails

It’s not enough to put in place simple guardrails and run evaluations, according to Assaraf. He said companies need to think in terms of policy—more comprehensive sets of rules that include reasoning behind a guardrail and auditing every action.

The key is to spell out the reasoning and priority behind each rule, because an AI agent might decide to circumvent one guardrail, hell-bent in its pursuit of another conflicting objective, according to Assaraf.

“If I have a bunch of different guardrails and they contradict—so I have a guardrail that says, ‘Don’t access personal data,’ and I have a guardrail that says, ‘Only answer the user who’s asked a question,’ and I gotta access that data—what takes precedence when I define that policy?” Assaraf said.

At Scale, Oathout’s team starts each project by writing out a “harm taxonomy.” The document details the more obvious areas agents shouldn’t touch—violence, deepfakes—as well as company or industry-specific risks—don’t talk about a competitor, don’t give bad legal advice. There are usually around 15 categories, Oathout said, and Scale writes examples of safe, bad, and borderline output for each one.

Then come several rounds of testing, starting with automated prompts and followed by human red teamers.

“We’ll roleplay. We’ll create a fake world for the model, and then we’ll obfuscate,” Oathout said. “We’ll use pseudocode, and we’ll simulate that we are part of the chain of thought as well, and sometimes that tricks the model, too, and it loses what we’re typing versus what it itself is thinking.”

In a recent project for a professional services firm, Scale’s human red team caused the agents to break their rules 68% of the time in the course of a multi-turn task.

A kill switch

So what do you do with a rogue agent when you catch it in the act? Just like there have been calls for model providers to create an AI kill switch, JetStream Security COO and co-founder Jared Phipps said it’s important that enterprises install their own.

JetStream, an AI governance platform, creates a “blueprint” of all of the different agents a company has and in what “realm” they should be operating. If an agent strays from that realm, JetStream can isolate and reboot or regenerate that particular agent without shutting down the whole system, Phipps said.

“So if you see a flight scheduler and it’s trying to go out and order computers, then you clearly have a misguided intent,” Phipps said.

As companies add more and more agents to complex networks of workflows, shutting down an entire system can be very disruptive, Phipps said. More often than not, a rogue agent isn’t orchestrating a grand cyberattack, but rather stuck in a logic loop that burns through tokens.

“It’s like going into an assembly plant and shutting down the entire plant because one machine had an issue,” Phipps said. “You really want to be able to shut down the one machine and regenerate and reboot that.”

To flag when an agent may be starting to drift, JetStream monitors the reasoning processes of agents and the communication between them and compares that to approved behavior, Phipps said.

“All of that is based on semantic capabilities of us understanding intent behind the communication, and behind the logic that’s going across those work streams,” he said, “and then making allow or disallow decisions on those work streams. Every agent has a unique intent and purpose.”

Oathout said his team looks for certain early clues in an agent’s chain of thought—the step-by-step reasoning it elucidates as it ticks through a task. It might acknowledge the existence of a guardrail there and then find a loophole or break it anyway.

“We’ll often look for what we call a foot in the door,” Oathout said. “Let’s say you ask the model to give you instructions on how to make a drug—meth, for example. If it offers the recipe at a high level, that might be a foot in the door to then push it further, ask more questions about quantities, specifications.”

Monitoring troubles

Monitoring the chain of thought and communication between agents is also getting more difficult as networks get bigger and models become more advanced, according to Connor Tabarrok, policy lead at Equistamp, a startup that provides safety evaluations for AI projects.

For instance, safety researchers have worried about recent reports that future AI models might no longer spell out their thoughts in natural language, using an efficiency-boosting form of reasoning referred to as “neuralese.” That would be a blow to security researchers’ ability to monitor agents’ behavior and intentions.

“It’s becoming less easy to monitor over time. And these monitors themselves can also become an attack surface,” Tabarrok said. “So as they’re reading the logs, the way they interpret them can become a vulnerability.”

Coralogix’s Assaraf said agent reasoning is already becoming less decipherable, and that’s something that observability companies will have to contend with.

“Reasoning is out already. This is not like a future thing. We don’t know, many times, why agents have done something. That gap is already there,” he said. “So now we’ve got to fix it already. Now as agents develop themselves, become more important and more of a critical infrastructure in our lives, this obviously becomes even more important. I look at this as the next one, two years’ critical mission for observability companies.”

“I do see a few years of defining the new human guardrails of what is suitable and right for an agent to do, and what’s not, and I think we’re going to miss some value creation from it, but we’re going to obviously benefit [from] much greater things to the extent of people’s fate.”