The Invisible Rationale

While attending to a patient with heart failure, a hospitalist receives an alert from an AI-enhanced clinical decision support tool suggesting a revision to a prescription. With a confidence score and a proposed change in dosage, the recommendation is displayed. What is not provided is any rationale for the recommendation. The clinician has no visibility into the data that informed the recommendation, what clinical evidence was considered, and how the model weighted potential competing risk factors. She is expected to justify a recommendation that she cannot trace, explain, or verify. The reasoning for the AI recommendation is hidden.

This is an AI explainability issue, and goes past a single case. With the increasing integration of AI in clinical practices, we see an even greater number of clinical decisions made with a lack of openness about how the AI arrived at its output. Health care systems are deploying these tools without the necessary technologies that allow clinicians to rationalize, defend, or explain to their patients what the AI is recommending.

"Black box" has become an industry term, especially for AI, and in clinical environments, it indicates a definable explainability issue. When an AI tool provides a recommendation without reasoning, clinicians are forced to rely on unverifiable, outside clinical judgment.

A recent study published in Nature Medicine investigates the explainability of AI technologies used in healthcare and found that most commercially developed diagnostic and predictive AI technologies do not explain their outputs. Most provide AI-generated recommendations or risk assessment scores without explaining what data inputs are involved, or the data inputs are unweighted variables in the model.

JAMA published research on the clinical impact of AI outputs that lack explainability. Outputs that lack explainability are less likely to be adopted by clinicians as part of their clinical decision-making on whether to follow the AI recommendation or not. Researchers found that the presence or absence of explainability created a binary response. Clinicians either accept the recommendation without any clinical reasoning, or reject the recommendation entirely. These responses indicate a lack of informed clinical judgment within the sphere of what the average standard of care would entail.

Opacity in AI systems has a direct and concerning impact on clinical documentation. Clinical documentation must explain either the recommendations or the rationale that led to a decision. Because that standard applies equally to AI-influenced decisions, clinicians must document the reasoning behind an AI recommendation even in the absence of explanation from the AI itself. This ultimately requires that the clinician provide a rationale based on an AI output without visible reasoning.

Georgetown University Law Center examines the legal risks associated with AI-driven clinical decisions and the lack of a documented rationale. They argue that the absence of explainable AI creates a gap that clinicians cannot document later. When a patient injury follows an AI-driven adverse clinical decision, the legal issue becomes whether the clinician acted reasonably. Without an AI rationale, the clinician is compelled to provide justification that is, in all likelihood, disconnected from the reality of the clinical encounter.

Although FDA has not issued formal explainability requirements for AI-enabled medical devices, the Agency has stated in position papers that clinical AI tools should offer clinicians a rationale to explain the clinical decisions. This is not yet a binding standard, and it presents an opportunity for health systems to implement these directives proactively and prepare for what is coming.

Pressure from policy and governing bodies has prompted attention to explainability in clinical AI. As part of developing health IT standards, ONC has incorporated the need for explainability so that clinicians can use AI tools responsibly.

Brookings Institution has investigated the policy landscape regarding the explainability of AI in healthcare. They found that various initiatives at the state and federal levels are revealing the expectation that AI and machine learning technologies used within the clinical infrastructure will provide some level of output explainability. Authors of the study concluded that the policy direction is evident even when there are no specific mandates and that health systems should view explainability as an institutional necessity rather than a vendor preference.

AMA has further supported this by stating that the explainability requirement should be a part of the policy health systems put in place regarding the acquisition of AI technologies. From this perspective, the evaluation criteria and the outcome of a clinical AI procurement should be justified in terms of the clinical documentation, patient communication, and defensibility of legal claims.

From a leadership perspective in health systems, the explainability problem must be addressed in three areas.

First, the criteria that govern procurement decisions should include explainability standards. In evaluating clinical AI systems, health systems must require that vendors explain and justify the output of the AI tool used, the level of explanation given to the clinician, and whether the explanation is part of the clinical workflow or available through an optional interface.

Second, documentation standards influenced by AI-driven clinical decisions must be updated. Clinicians should be able to document and provide rationale for clinical decisions that include AI output through structured fields provided in an electronic health record system. Free text notes are inadequate for this documentation because they lead to an audit trail that is unreliable and lacking in sufficient detail.

Third, health systems must implement a minimum standard for AI explainability across all clinical AI tools deployed within the enterprise. This standard should define what constitutes adequate reasoning from the tool, how that reasoning must be presented to the clinician, and what documentation is required for the clinical record when the AI output lacks sufficient explanation.

Institutions that treat explainability as a core requirement will be positioned to support their clinicians and defend AI-influenced decisions in legal and patient safety reviews. Those that continue to deploy opaque tools will find that the invisible rationale behind the AI is the most visible problem when a question arises.

Nature Medicine has examined explainability characteristics of clinical AI tools. JAMA has published research on the clinical impacts of opaque AI outputs. Georgetown University Law Center has analyzed the legal risks of AI decisions lacking documented rationale. FDA has recognized the explainability problem for AI-enabled medical devices. ONC has incorporated explainability into health IT standards development. Brookings Institution has researched the policy landscape for AI explainability in healthcare. AMA has recommended explainability requirements in AI procurement. This edition connects to the documentation and clinical decision themes in Editions AE, AF, and AI of this newsletter.

Christopher Hutchins
Founder & CEO, Hutchins Data Strategy Consultants