Responsibility in Practice

Related on The Signal Room: Responsible AI in Healthcare with Asha Mahesh. Related from HDSC: Responsible AI in Healthcare.A regional health network has begun utilizing an AI-driven tool for clinical decision support to make suggestions for early intervention with patients perceived to be at risk of being readmitted within 30 days of being discharged. The tool evaluates and identifies a patient as being low risk. The care team adheres to the advice and subsequently does not arrange for a post-discharge follow-up visit. The patient is readmitted nine days later with issues that a post-discharge follow-up visit would have very likely identified. The treating physician poses a direct inquiry: who holds the liability for the decision taken based on the model predictions?

No one is certain. The model was constructed by the vendor. The model was activated by the IT squad. The clinical leadership sanctioned the use case. The physician responded to the model prediction. The patient faced the repercussions. Responsibility is, in this case, so thinly spread that it in fact belongs to no one. The lack of ownership is one of the primary issues with AI in healthcare. Technology, in this example, adds an extra element to the clinical decision-making process and the institutional frameworks that determine who is answerable have not evolved.

The clinical practice has a rigid structure of who is responsible for what. A test is ordered by a physician who is then responsible for reviewing the results and making a treatment decision. If something goes wrong, the decision by the physician is the traceable point of liability. This changes when AI is integrated into the practice. The decisions become less clear. The AI will make a recommendation based on some data. The clinician may or may not want to review the rationale behind the recommendation. The recommendation may or may not be communicated clearly, and will almost certainly lack an accompanying qualitative assessment of certainty. The institution may or may not have communicated what, if anything, is expected of the clinician in response to the recommendation.

Per the American Medical Association, stated in writing, at the end of the day, the clinical decision must be made by the physician, regardless of the AI tools utilized. This is easy to enforce in writing, it is difficult to implement in standard practice. A study in the JAMA Network Open found that, deferring to an algorithm recommendation, even when the clinician has clinical reasoning that goes against the recommendation, is a common practice when the recommendation is accompanied by a confident score.

In the BMJ, there was a study on the introduction of artificial intelligence into diagnostics. The introduction of AI tools resulted in clinicians modifying their initial diagnoses to match the AI suggestion. The authors named it automation bias. This indicates that the impact AI has on clinical decision-making is disproportionate to the role it plays.

Most health systems lack either internal policies or frameworks outlining responsibility for adverse outcomes stemming from AI-assisted clinical decision-making. The Inspector General Office has explored cases involving algorithmic tools for Medicare Advantage that resulted in coverage denials. These denials were eventually overturned through the appeals process. In these cases, ownership of the denial, be it the algorithm, the payer, or the peer reviewer, remained unanswered.

The National Academy of Medicine has acknowledged, and in such instances, formally recommended, health systems and policy makers construct institutional frameworks that articulate the intersection of AI tools and clinical decision-making processes and the ownership of outputs. Those policy documents proposed the creation of encumbered stewardship of models, defined decision-making frameworks pertaining to the outputs of models, and, in such, mechanisms that require documenting the use of AI in the final clinical decision.

Health Affairs published the first study of AI-assisted clinical decision-making and the liability landscape. One of the primary findings from this study was that malpractice law is applied to a clinical scenario where the clinician is the ultimate decision-maker. The introduction of AI as a parallel decision-maker creates a gap in the intersection of legal frameworks, and, in the event of negative outcomes, that gap results in a lack of clarity regarding liability.

Structural design goes past policy statements when referring to the design of responsibility in AI-enabled healthcare. As one part of that design, healthcare systems should establish a decision-making structure that defines, for all AI use cases, what the model will do, what the clinician will do with the model output, and what level of documentation is required at the point of action. Coupled with this, there needs to be a system of monitoring the eventual use of model recommendations, including overrides, concordance to outcomes, and the persistence of automation bias.

The Joint Commission has highlighted the incorporation of clinical decision support systems as a pivotal element of patient safety in health systems. Their standards mandate healthcare institutions to assess the impact of decision support systems on clinical outcomes and to have processes in place to identify when those systems result in adverse outcomes.

Studies conducted at Brigham and Women Hospital have explored the structured use of overrides for AI recommendations in clinical practice. Their findings suggest that institutions with structured processes for documenting and reviewing overrides have a greater ability to recognize when a model is producing poor quality outputs, and to capture the right people when those outputs cause harm.

Health system leaders need to consider whether AI workflows have been designed with the same care as other clinical processes when it comes to ownership and responsibility. If the answer is no, the missing element is not technology. It is a leadership issue, and addressing it requires the same operational discipline health systems apply to credentialing, measuring quality, and ensuring patient safety.

There is literature from the AMA on physician responsibility in AI-driven decision-making, and JAMA Network Open has published work on clinician trust in algorithms. The BMJ has looked at automation bias in the diagnostic process. The OIG has studied algorithmic denial of coverage in Medicare Advantage. The National Academy of Medicine has published work on the creation of institutional policies for AI in clinical workflows. Health Affairs has looked at the risk of liability in the context of AI-driven decisions. The Joint Commission has addressed patient safety and decision-support technology. Research coming from Brigham and Women Hospital has studied the structured override process. This edition relates to the clinical ownership and design topics from Editions N, T, and U of this newsletter.

Christopher Hutchins
Founder & CEO, Hutchins Data Strategy Consultants