AI cyber threats require oversight, guidance

Sponsored by

Your AI strategy has a security problem: 43% of IT leaders now cite security concerns as the top barrier to expanding AI agent deployment, according to research from JumpCloud. Before giving agents more access to apps, data, and workflows, make sure you can govern the identities behind them. See how to secure every identity, human or not.

Keep an eye on things, especially when it comes to AI and your IT stack.

Monitoring AI agents is becoming more important as IT pros manage the technology’s risk and security impact.

For Amr Ali, WitnessAI head of machine learning, challenges related to AI assistants operating within systems—and the potential ramifications of improper guardrails and restrictions—has become a pressing concern.

“Usage of language models within workflows, within day-to-day operations, makes growth much faster, workflows much more efficient, and this means you have multiple lines of usage,” Ali said, adding, “The radius of something going wrong could go very wide.”

Looking in. In order to manage that expanding risk, visibility into these AI tools is extremely important. IT pros should take a two-part approach to observability, Neo4j CTO Phillip Rathle said.

Outside, often third-party-led moderation allows for visibility into decision processes. That can lead to better understanding of where information is going and how it’s being managed—and also result in automated monitoring that feeds into self-improvement loops.

The second step is internal monitoring.

“You want a feedback loop to the degree you want to minimize human input and human work, and in both building but also in supervising—spending a lot of time babysitting the AI and being a human in the loop—that can get very expensive,” Rathle told IT Brew.

Rathle added that judgement, which he defined as human oversight to ensure the data AI models or processes use in operations is legal, ethical, and secure, is also important. Oversight is important here as well, though it’s an open question whether it can catch any potential maladaptive information.

“If you train your model on data, that data becomes fair game…no matter how much monitoring you have on the outside, what you have on the inside and what technologies actually execute your AI decision has a huge bearing on the output,” Rathle said.

Pencils down. Control of data should be another focus, Ali said, to help prevent leakage. Whether the personally identifiable information (PII) is coming from spreadsheets or is otherwise available in a database for LLMs to grab, there need to be parameters of control.

“This is where the guardrail comes in,” Ali said. “This is where you would need a way to filter this PII from leaving the local laptop and before arriving at the cloud-based language model.”

Reasoning processes are now so complicated that tracking them can be a difficult task, one that is swiftly being outstripped by model capabilities to redirect oversight. The key, Ali told IT Brew, is to focus on model action points.

“If it’s an agent, you work on the output fronts, the tool calls, the MCPs, all the components this language model interacts with, and this is the easiest way to govern the usage of these AI assistants and how these agents operate,” Ali said.