Inside Microsoft’s latest open-source AI vulnerability tooling

Why wait until an AI system is done being built to test its vulnerabilities?

Ram Shankar Siva Kumar, a data cowboy (yes, that’s his official title) at Microsoft, leads the team responsible for RAMPART, an open-source tool that red teams and other cybersecurity pros can use to test AI agents’ vulnerabilities.

“The reason why we have tooling investments is it’s really hard to do all these things manually,” Siva Kumar said.

The team is also working on Clarity, another open-source tool that offers feedback to developers writing code.

Running RAMPART. The tool, which is based on top of Microsoft’s PyRIT cybersecurity automation framework, injects red team tools into developer workflows. For example, if a developer is building an AI agent, they can use RAMPART as a tool to simulate prompt injections.

In an email, Siva Kumar said RAMPART allows red teamers to vary their simulated attacks on AI agent environments, as well as convert red-team scenarios into “repeatable tests that engineers can run in continuous integration, pre-release validation, and ongoing safety regression testing.”

RAMPART also repeats the attack scenario to see if it can bypass any guardrails associated with an AI model. “You can take a prompt, you can mutate it, you can generate thousands of variations and then you can repeat those tests multiple times, because here’s the thing: Your agent may deny you on the first time, may deny on the second time, but the third time it may pass, and attackers are going to do the same thing,” Siva Kumar said.

Give me some Clarity. The other new open-source tool, Clarity, is meant to run as either a web UI, a direct embed into a coding agent, or a desktop app. Its AI attempts to ask questions that those outside of the direct build pipeline (i.e., safety engineers, product managers, and others) would want answers to, according to Microsoft’s announcement.

Siva Kumar wrote that while Clarity is able to help with software projects, it isn’t focused on writing code.

“Even the best coding-focused LLMs can help you think about code, but the assumption is that you already know what you want to build,” Siva Kumar said. “Clarity agent helps you think through what you’re building by talking to you about the problem.”

Hatem Ayad, CTO at Clarvos and a Microsoft veteran, said that Clarity is different from other Big Tech companies’ offerings because it offers an end-to-end framework for developers.

“Microsoft is very good in putting together frameworks for big projects or big initiatives, and the AI agent tech workflow now is very much the standard every company is trying to use, so how to build agents, how to deploy agents, how to find security issues, and then how you patch agents,” Ayad said.

Why is Microsoft doing this? Ayad said that when Microsoft offers an open-source tool, they are trying to build a community around it, similar to the way that the company contributed to OpenTelemetry.

“Microsoft’s trying to become a reference architecture for AI safety, which is a gap right now,” Ayad said. “But you have to be close to Microsoft’s ecosystem to use RAMPART, so they are trying to build an edge or niche for themselves.”