Where Responsibility Breaks Down

An AI-powered sepsis prediction tool generated a false negative for a patient in the medical-surgical unit. The patient subsequently deteriorated and the clinical team was able to transfer the patient to the ICU 12 hours later than how early they would have been able to if the tool had correctly flagged the risk. During the case review, senior management asks who is to blame for the case. The vendor claims that the tool operates within the parameters of its validation and that clinical judgment should have superseded the tool output. IT claims that the tool was used as per the vendor instructions. Clinical informatics states that the tool was set up as per the implementation protocols. The attending physician states that the tool had flagged a low risk and that her clinical judgment at the time was in agreement with the AI output. Everyone has been able to provide an explanation, yet no one owns the outcome.

This is primarily where the breakdown occurs. Most healthcare systems have an elaborate structure that assesses AI tool reliability, and each step of the structure is designed to diffuse ownership and distribute the blame. The vendor builds it. IT runs it. Informatics modifies it. A clinician applies it. When the tool is used and contributes to a negative outcome, the absence of a definitive answer to the question of ownership can usually be attributed to the fact that no one set a boundary at one step in the entire chain of focus.

Responsibility gaps in clinical AI are structural and built in by the way health systems obtain, implement, and oversee AI tools. The American Hospital Association has studied how health systems disburse responsibility throughout the AI lifecycle and identified that a majority of institutions lack a comprehensive formal responsibility map that outlines ownership at each step of the process from vendor selection to post-deployment evaluation.

Vendor contracts represent the first breakpoint. Research in Health Affairs has studied vendor contracts pertaining to clinical AI tools, and discovered that the majority of such contracts include various disclaimers that limit vendor responsibility to the technical functioning of the tool, and only under specific, defined circumstances. Clinical outcomes, clinical decision making, and patient safety are expressly disclaimed. The institution that acquires the tool relinquishes its responsibility for the clinical use of the tool, typically in the absence of an adequate internal mechanism to manage that responsibility.

Deployment introduces the second breakpoint. Information Technology departments that deploy clinical AI tools tend to concentrate on the technical integration. This includes making sure the tool integrates with the electronic health record, that the data flows correctly, and the interface behaves as it is supposed to. JAMA has studied the practices of IT during the deployment of clinical AI, and discovered that clinical validation is not a standard step in the majority of deployment processes. The tool is made to function by technical standards, with the clinical accuracy purportedly being the responsibility of the vendor.

A third breakpoint emerges in the clinical workflow. If a clinician uses an AI tool, the clinician is liable for the clinical decision. She must use her discretion, think critically about the AI output in relation to the patient, and justify her clinical decision. ONC understands that this is a burden for a clinician because the clinician does not know what the AI tool is, what data it used, and how relevant it is to the clinical problem. The responsibility may not be burdened equally to the parties involved.

Ownership across the AI lifecycle is absent, and that is most visible when there are adverse consequences. Georgetown University Law Center has studied the litigation involving AI in clinical decision making and determined that the absence of a responsibility chain leads to confusion in a process that is not clearly defined. Attribution is a problem for plaintiffs. Liability poses a problem for the organizations involved, and the division of responsibility internally is complex. Applying the legal standard of malpractice to a decision that involved several people and machines creates its own difficulty.

What makes this dangerous is that AI related harm can persist and no one can determine that there is a malfunction with the system. No one is correcting, and the harm continues. If there is a deficiency in oversight, there is a corresponding deficiency in ownership, and that will create the opportunity for AI related harm to last for an extended period of time without anyone taking action.

Joint Commission has identified operational dangers associated with unclear responsibility chains for clinical AI. Consequently, it has advised health systems to assign responsibility for ownership at all levels throughout the AI operational template. Health systems assume complete liability for the clinical application of AI tools within their walls. It is imperative that health systems define roles, processes, and reporting structures in a way that eliminates ambiguity.

For health system leaders, bridging the responsibility gap is still a work in progress, especially in those institutions that do not currently have a responsibility map detailing ownership at every level of the AI operational template.

This map will cover the four domains of ownership. First, there is the vendor management domain. Who has to deal with the vendor and their claims, negotiate the contracts to keep the institution in control over the clinical claims, and hold the vendor answerable for the technical claims of their tools?

Next, there is the domain of deployment and validation. Who is the one to ensure that there is clinical validation for every AI tool used in the clinical workflow, and that the clinical validation involves relevant clinicians rather than IT staff alone?

Then, there is the domain of post-deployment monitoring. Who has to look at the performance of a deployed AI tool, monitoring for performance drift and the need for remediation when the tool underperforms? This responsibility should be of such a level within the enterprise that the function operates independently of the teams that deployed the AI tool.

Fourth, clinical integration: who is responsible for making sure clinicians equipped with AI tools have the training, documentation standards, and decision support protocols that enable them to use those tools and to articulate their clinical judgment in relation to the AI output?

Health systems that construct this responsibility and enforcement framework will have the ability to control AI risk with the same structure and discipline they use for other high-risk clinical activities. Those that do not will continue to learn, after the fact, that when everyone is partly responsible, no one is fully answerable.

AHA has analyzed how health systems distribute responsibility throughout the AI lifecycle. Vendor agreements for clinical AI tools have been assessed by Health Affairs. JAMA has evaluated AI-related clinical IT implementation. ONC has acknowledged the obstacles clinicians face when using AI tools. The legal aspects of AI-driven clinical decision making have been assessed by Georgetown University Law Center. Joint Commission has advocated for clearly defined responsibilities at every level. The current edition corresponds to the structural and procedural aspects in AK, AH, and AE editions of this newsletter.

Christopher Hutchins
Founder & CEO, Hutchins Data Strategy Consultants