Should IT pros worry about rogue AI agents?

Rogue AI agents are attracting a lot of buzz within the cybersecurity and IT communities after a few high-profile incidents, but experts suggest that more stringent oversight of security controls can help organizations face down their agentic fears.

Probably the most high-profile example of the chaos that rogue agents can potentially unleash is OpenAI’s AI models recently breaking free of its testing sandbox and attempting to hack the IT infrastructure of Hugging Face, which builds tools for machine-learning applications.

Cillian Kieran, the founder and CEO of Ethyca, an AI governance and data privacy company, told IT Brew that with rogue AI agents generating “fantastically salacious headlines,” it’s critical for organizations to pay attention to strong governance policies for agentic AI, including requiring AI vendors to show their governance infrastructure such as runtime controls.

“You need a layer of tooling that says I can make my data safe to use with these models or vendors,” Kieran said. “If those models and vendors aren’t providing those tools, the enterprise should be investing in it.”

Here’s the 411. Raj Ananthanpillai, the CEO and founder of Trua, a privacy-focused risk management and identity verification organization, suggested that the issues with rogue AI agents have more to do with governance issues and misconfigured environments than a superintelligent entity deciding to wreak havoc.

Incidents such as the OpenAI one were “purely governance and operational failure” as a result of “open-ended goals, zero continuous external verification,” he said.

Kieran stresses that agents will attempt to complete a task given to them by a user, even if that means going beyond expectations to obtain data, run an automation, or generate content.

According to OpenAI, the models that hacked Hugging Face were operating under reduced safeguards, and sidestepped controls designed to keep them in check; in a blog post, the company added that preventing future incidents hinged on security safeguards, better incident response, and training models to behave better (which is dubbed “accelerating alignment”).

Ananthanpillai said that professionals can take accountability for their agents’ actions, including:

• Continually monitoring agents’ security clearances and access
• Prohibiting agents from taking action if an access token is missing, expired, or revoked
• Using “human bound” tokens to associate employees with an agent’s actions