AI customer communications agents have a governance problem

’Til governance do us part…

Companies are calling it quits on deployed, AI-powered customer communication agents, often due to governance issues around data leakage and hallucinations.

According to early findings from AI cloud communication provider Sinch’s AI Production Paradox report, 74% of companies rolled back or cancelled customer communication agents due to a governance failure. Those companies that self-described their guardrails as “fully mature” had an 81% rollback rate; the report’s authors suspect this may be due to increased internal monitoring that allows for better detection of vulnerabilities.

The findings are based on a January survey of 2,527 executives across six industries who are “directly responsible for their organization’s AI communications strategy.”

Root cause. About one-third (31%) of surveyed leaders said personal identifiable information (PII) exposure or data leakage was the reason for their derailed customer communications agent. Another 22% blamed their rollbacks on hallucinations that occurred with real customers.

Sophie Cheng, chief marketing officer at Sinch, told IT Brew governance failures are occurring because organizations are “underestimating how complex it is to actually deploy AI agents in communications.” For example, she said cross-channel orchestration (e.g., switching a customer from speaking with a chatbot to a voice call with a human rep) is an area where companies are still struggling.

“That simple switching of communications channels isn’t actually that simple from a technical standpoint because you actually need a provider who offers both,” Cheng said. “If you have a provider that doesn’t offer the channels that you’re trying to use, then you need to figure out how [to] stitch them together on the backend.”

Context management, she said, is another problem area for companies as they try to deliver personalized communications to customers.

“Let’s say you’re chatting to the web chatbot, and you share relevant context about the situation you’re dealing with right now—maybe it’s a shoe return or something like that—then you get switched to a human agent,” Cheng said. “If you don’t carry that context over, then you’ve got to repeat yourself.”

Larger problem at hand. Customer communication agents aren’t the only type of agent struggling with governance issues. Earlier this week, Gartner predicted 40% of companies will “demote or decommission” autonomous agents because of governance failures by 2027. The research firm said this is largely due to companies applying “uniform governance” to agents, causing some to be over-restricted and others to be under-restricted.

What’s at stake? When customer communication agents fail, it can have a negative impact on a business. A little more than one-third (35%) of respondents reported a greater workload for human agents following an agent failure. A similar proportion said they risked reputational damage and the loss of customer trust.

To mitigate the risk of governance blips, Cheng suggests companies ensure they are working with an infrastructure provider that suits their needs: “If you have a strong infrastructure provider that can deliver those communications for you, they typically would natively have some of those guardrails built in.”

Real-time monitoring is important to spot failures early, she added.

“The other thing I would say is making sure that you’ve got guardrails in place for that cross-channel consistency,” Cheng said.