Healthcare AI accountability shifts to hospitals as rules remain fragmented

Healthcare AI accountability is becoming a more immediate legal and operational issue for hospitals as artificial intelligence moves into clinical support, patient messaging, claims administration and health-data exchange. Alaap Shah, a member of law firm Epstein Becker Green, told Competition Policy International, a PYMNTS company, that fragmented regulation does not leave providers free of exposure.

Shah said healthcare organizations are entering a phase in which the central issue is no longer whether AI tools function, but who must answer when they produce flawed or disputed results. Federal agencies are reassessing some oversight approaches, states are developing their own rules and older statutes are being applied to technologies those laws did not specifically contemplate, according to Shah.

“Self-governance matters because defensibility matters,” Shah told CPI, adding that risk is already appearing in healthcare AI use.

Who is accountable for healthcare AI?

Shah said hospitals and healthcare companies cannot assume they may wait for a single AI statute or a unified federal rulebook. Existing laws on privacy, discrimination, consumer protection, contracts and professional duties can apply to AI-enabled conduct even when those laws do not name artificial intelligence.

That means a patient-facing model that delivers different outputs across demographic groups could raise discrimination concerns, while a tool that affects a physician’s clinical judgment could become relevant in a malpractice dispute. Shah said AI is entering healthcare’s existing liability framework, rather than sitting outside it.

The Food and Drug Administration continues to oversee some software as a medical device, but Shah said the boundary between regulated and unregulated tools remains unclear in some cases. A company may describe a product as decision support, while regulators or state medical boards may see the system as crossing into clinical practice or medical decision-making.

Shah said the sector should expect regulatory fragmentation to persist for some time, with federal agencies and state authorities applying overlapping standards.

Why vendor contracts are becoming part of AI governance

Hospitals often deploy AI systems built and operated by outside vendors. Shah said that creates a practical accountability problem: the vendor may control model logs, training information and performance data, while the hospital may still have to explain what occurred if the system contributes to a disputed decision.

For that reason, Shah said contracts are becoming central to healthcare AI governance. He pointed to the need for logging obligations, record preservation, audit rights and defined access to evidence generated by AI processing.

“To the extent that any events could be logged in the AI processing, that is something that needs to be happening so we can understand how that AI operated and why the input led to the output,” Shah said.

The mechanism is straightforward: if an AI tool influences a recommendation, denial, message or clinical workflow, an organization may need a record showing what data entered the system, how the system processed it and what output it produced. Without those records, a clinician, hospital or insurer may struggle to reconstruct the basis for a disputed outcome.

How AI complicates privacy and bias controls

Shah also said AI adds pressure to healthcare privacy practices, including de-identification. Removing names and other direct identifiers can reduce risk, but AI systems may be able to combine data sets, infer attributes or reconnect information that seemed anonymous when viewed separately.

“It’s a real possibility that AI algorithms could re-identify individuals if sufficient data gets put in, even if de-identified in the first instance,” Shah said.

Bias is another area of exposure. AI tools can affect treatment decisions, prior authorization, insurance coverage and patient communications. Shah said organizations may face litigation, regulatory scrutiny or reputational harm when systems produce uneven outcomes for protected groups, regardless of shifting federal enforcement priorities.

Shah said healthcare organizations are responding by building inventories of AI tools, classifying systems by risk, assigning accountable owners, training staff, documenting controls and allocating responsibilities with vendors. In his view, the aim is to create a defensible record showing that an organization acted responsibly if an AI-related failure is later challenged.

This story draws on original reporting from PYMNTS.