AI threats and risk are changing how we see the technology—what does that mean for IT pros?

A long-simmering sense of public mistrust in AI finally boiled over this month, leading the tech industry to recalculate how it builds the technology—although much of the danger may stem from decisions about deployment and safeguards.

After researcher Jacob Coxon publicly quit his position at Anthropic on September 8, warning that the company was being irresponsible in its stewardship of its AI models and careless with the danger they pose, fears that AI could pose an existential threat to humanity overtook the public discourse.

As Uncle Ben told Peter Parker, with great power comes great responsibility. That’s a potential lesson for an AI industry deploying autonomous agents without proper guardrails, said Ramy Rahman, senior solutions engineer at ArmorCode, and it needs to change.

“The challenge now is we really need to up our game when it comes to extending the right amount of privilege to the AI and holding its hand through the process, which turns out to be extremely difficult when you have something that is solving mathematical problems that are at speed,” Rahman said. “Humans are not capturing the risks quickly enough.”

Change is coming. Politicians in Washington are moving to restrain AI companies, and public trust in the industry is low. CEOs like OpenAI’s Sam Altman and Anthropic’s Dario Amodei are calling for a slowdown in model development—though it’s unclear how exactly this would happen—along with third party (but not necessarily governmental) oversight.

To Aya Ibrahim, senior fellow for economic and national security at the AI Now Institute, corporate calls for pacing and third-party controls are more a ditching of responsibility than anything else. For IT pros, that means paying close attention to how AI companies are managing the risk, especially when teams are expected to continue to work with their products.

“It requires certain safeguards, it requires certain data practices, certain hygiene, cybersecurity practices, protections, access control, all of that,” Ibrahim said. “A lot of the stories that we have seen over the past few weeks would suggest that all of those things I just named are apparently an afterthought to many people building the technology.”

Threat detected. The level of overall threat from these models and agents is debatable, with some experts telling the public they believe AI will lead to a 10% or higher danger of human extinction in the next decade. More skeptical voices are urging caution while acknowledging the longstanding risks presented by powerful models.

Warnings should be taken with a grain of salt, said Santa Clara University Professor Ram Bala, whose work focuses on AI and the AI industry. Bala told IT Brew that the “extinction-level risks” cited by Coxon and others are overblown, and that Amodei’s warnings are more about the power of the models.

“The capabilities of these systems have been on an upward swing—I can see the trajectory, compared to a year ago or two years ago,” Bala said. “It’s not only the other capabilities increasing…my own experience with working with systems like these at scale is that they provide a lot of benefits, but they also need to be reliable and they raise the risk level of bad things happening.”

A lack of proper controls at the top level doesn’t mean that IT pros should stop using the models altogether, Rahman said, in part because staff will reject a total shutdown. Instead, they need to be careful and to institute necessary restrictions on the ground.

“You can partially stop some usage, but I think you’re going to find that you’ll get a lot more pushback there,” Rahman told IT Brew. “We need to be able to get guardrails and administration of it more finely tuned and controlled so that people can adopt it safely.”