OpenAI releases disruption report for malicious uses of AI

OpenAI is expecting to see more attackers use ChatGPT and other AI models to produce malicious code and tools, according to a new report. However, the company sees no evidence that attackers are using its AI models to produce novel offensive capabilities.

In its report, OpenAI outlined several instances where it disrupted bad actors, including nation-state actors and criminal groups, who were attempting to use ChatGPT’s capabilities to produce malware or engage in other nefarious activities against what the company called “the democratic principles America has always stood for.”

“[Promoting democratic AI] includes preventing the use of AI tools by authoritarian regimes to amass power and control their citizens, or to threaten or coerce other states; as well as activities that may result in society-scale harms such as malicious cyber activity, organized crime and scams, and covert influence operations,” the report stated.

OpenAI pointed to its submission to the White House Office of Science and Technology Policy’s AI Action Plan as evidence of the company’s commitment to building and maintaining “democratic AI.”

OpenAI did not respond to a request for comment in time for publication.

That’s not how you use that. The company reportedly found individuals linked to People’s Republic of China (PRC) government entities using ChatGPT to help with “development, profiling, and bureaucratic functions.”

Specifically, some of those accounts were trying to use ChatGPT for large-scale monitoring activities, such as analyzing datasets gathered from social media platforms. OpenAI reported users asked the model to help design tools and generate promotional materials, but hadn’t put actual monitoring into practice.

“For example, we recently banned a user, possibly using a VPN to access our services from China, who was asking ChatGPT to help design promotional materials and project plans for a social media listening tool, purportedly for use by a government client,” the report said.

OpenAI described its disruption tactics as banning users who appeared to be linked to PRCe government entities and seemed to be using ChatGPT for “more bespoke” targeted profiling and online research. According to the report, the models only returned publicly available information instead of sensitive details when utilized by political actors.

Not the vibe. In another case study, OpenAI pointed to a Russian-language malware tool developed through vibe coding. The company disrupted the use of the tool, which it linked to Russian-speaking criminal groups.

According to the report, OpenAI discovered accounts trying to use ChatGPT to develop and refine malicious tools that included credential stealers, a remote-access trojan, and features to evade detection. With safeguards in place, the models “refused requests that were clearly malicious and lacked any legitimate application,” and didn’t execute any threat actor tooling and workflows.

What else is being done? OpenAI reported that it interrupted threat actors who seem to be using various AI models for recidivism, or repeated harmful activity, and phishing.

The company reported individuals involved in these harmful activities have been adapting their behavior, including scrubbing signs of AI usage from their content—specifically manually removing the use of em-dashes before publication.

Zohaib Ahmed, CEO of deepfake detection platform Resemble.ai, told IT Brew in an email that OpenAI’s approach to preventing misuse is at the model level, and the industry still needs additional detection capabilities.

“That’s important, but the report shows bad actors are using multiple AI providers and hopping between models when they get banned,” Ahmed wrote. “This is why detection infrastructure is critical—you can’t rely on model-level restrictions alone.”