Research on AI jailbreakers reveals where they go to sell their wares

Getting past guardrails on a road? Bad. Getting past guardrails for AI models? Lucrative for AI jailbreakers—provided they know where to sell whatever data they steal.

Open Measures, a threat intelligence resource focused on attacker search across platforms, recently published research on the increased sophistication of AI-infused cybercrime—and where AI jailbreakers and other attackers are congregating. The report shows how vast the jailbreaking ecosystem is, spread across a number of sites, primarily 4chan, Discord, and Telegram.

Report author Ryan Aliapoulios, a researcher with Open Measures, told IT Brew that the space has evolved for cybercriminals.

“AI jailbreaking, at one point, was a niche community interest,” Aliapoulios said. “Now it’s become a cross-platform ecosystem that has its own division of labor, where each social platform has its own role in the ecosystem.”

List ’em. 4chan operates as a discoverability platform, offering users access to jailbreaking tactics and technologies. Discord is where jailbreaking evolves into collaborative R&D. Threat actors go to Telegram to sell their wares.

Telegram has been a vector for threat actors for some time. In March, Prasad Tharippala, Asia–Pacific field CISO at Versa Networks, told IT Brew how he tracked down an attacker selling stolen data on the site.

“At one of the conversations, he started showing me the dashboard of that particular company,” Tharippala said. “And that is where we finally concluded, ‘Okay, this is the point where the data is getting leaked.’”

Expanding the zone. X, the site formerly known as Twitter, has also become an important part of the story, Aliapoulios told IT Brew, serving as a promotional bazaar.

“There’s a lot of AI content that gets posted, a lot of new technologies, new apps being built every day—it’s kind of hard to tell if you’re an average user of that platform,” Aliapoulios said, adding, “This is more a subjective interpretation than a data one, but it’s a gateway to other places, because they often link to Discord, or they link to two places where people are actually workshopping new AI apps and techniques.”

One of the reasons AI threat actors are hard to track online is the thin line between criminal and unethical use of the technology—simply breaking LLM policies can be safe and legal even though it opens the door to other misbehavior. The problem is part of an overall back and forth on AI ethics that has driven the tech industry’s talent hiring concerns.

“We try to pay attention more to the fringe communities on fringe platforms to see where these things go, because it seems like they keep spreading outward,” Aliapoulios said.