How to prepare for a healthcare AI audit

Kevin Dunnahoo often works with CIOs, CISOs, CFOs, and other healthcare C-suite executives on internal risk assessments.

A quarterly evaluation takes on specific objectives, like identity and access management, cybersecurity evaluation, or, lately, what’s going on with all this AI?

While HIPAA regulations require a continuous risk assessment of how an organization manages its protected health information (PHI), a healthcare organization is more likely to face an audit from its board, according to Dunnahoo, director of healthcare technology, cybersecurity, AI, and IT audit at global consultancy Protiviti.

“Many of the boards are asking flat out, ‘Hey, I want to understand what AI we’re using and how we’re using it, and how are we controlling it,’” Dunnahoo told us. “They want a true AI governance-focused review on the plan, on an annual basis at this point, because they understand that there is so much potential risk.”

Dunnahoo and other healthcare IT pros shared how companies can deploy AI in a way that satisfies governance standards, and, hopefully, any auditors.

Listen up. An Eliciting Insights survey of 120 US healthcare leaders, published in 2026, found that 3 out of 4 respondents use or plan to use at least one AI application—an increase from 59% in 2025. Top use cases include clinical notetaking and ambient listening, clinical documentation improvement (CDI), and coding.

A company introducing an AI deployment—ambient listening devices, for example—needs to answer some key questions, according to Dunnahoo. First, was the business case documented and approved? There should be sign-offs from an AI governance committee, and the organization should consider individual security reviews, Dunnahoo added: In addition, the business owner (not the CIO or CISO) should sign off and own the AI use; the committee then signs that it has reviewed and approved the risk.

Other questions to answer include:

• What data will the tool have access to?
• Who can access the AI?
• What is the value expected? (“ROI is a real question that’s being asked in a lot of these situations. And is that ROI commensurate with the potential risk that we’re also bringing into our organization to use that AI?” Dunnahoo said.)
• What will be the monitoring mechanism?

MultiCare Healthcare System, a healthcare network in the Pacific Northwest, recently announced its use of ambient listening in some of its 13 hospitals. CISO Jason Elrod (and executive advisor at cybersecurity company Elisity) shared how it works: a clinician asks the patient’s permission to turn on the recording device, and the tech generates draft notes from the conversation and surfaces relevant clinical details.

But that kind of system requires guardrails, and “ambient” doesn’t mean “always on,” according to Elrod; the system automatically deactivates at the end of a meeting, or when the clinicians leaves a patient’s chart, for example.

“I’m less interested in whether something has an AI label on it than I am in what data it can see, what systems it can touch, and what actions it can take,” Elrod said.

How to audit agents. If a board member or even a HIPAA auditor arrives to investigate a data breach, healthcare IT pros should be ready with receipts like third-party risk assessments and documentation revealing configuration decisions for AI agents, according to Ryan Meehan, managing director and healthcare practice leader at compliance-services firm Schellman.

As organizations deploy agentic AI—say, for appointment automations—Meehan recommends additional safeguards like:

• Giving agents unique IDs, with least privilege (i.e., having access to only essential resources to perform a task)
• Robust audit logging so an IT pro can reconstruct an event sequence if necessary
• Knowing your “source of truth,” and exactly where your AI is pulling data from
• Constrain input possibilities, tailored to a given application

“I think it goes back to, ‘Did you build it in a way that had trust with all the parties involved?’” Meehan said.

Meehan recommends frameworks like ISO 42001, NIST’s AI Risk Management Framework, and especially agent standard AIUC-1 for technical controls.

In the shadows. From Dunnahoo’s perspective, the biggest problem these days is shadow AI. Companies like ServiceNow and Palo Alto Networks have introduced products that look for network signatures of AI usage, such as a clinician using an unapproved chatbot.

But if a doctor uses their phone as a hotspot, with their camera-equipped, AI-enabled Meta glasses connected to it, many tools can’t flag that kind of activity, according to Dunnahoo. How to manage the rogue AI usage remains a main concern on the minds of many healthcare leaders.

“That is the million-dollar question, and it’s one everybody is trying to answer right now,” Dunnahoo said. “From an audit, we are trying to look for how the organization is keeping a pulse where that shadow AI would be occurring. We’re looking at how the organization is communicating and setting that policy and expectation for what is allowed AI.”