Accountability has not changed. What has changed is that the rationale behind a clinical decision can now be generated by a system that does not explain itself. WHAT HAS ACTUALLY CHANGED What has not changed is the accountability and responsibility clinicians have for the decisions they make and the records they sign. What has changed is the variety and types of new content that informs those decisions and populates those records. In my years of experience with healthcare data and analytics, the predominant focus was data integrity. Was the data accurate? Was the record complete? Accuracy and completeness of records were the critical elements of data integrity, but now the focus has changed. It is not about the record or the data, but rather the content that is produced by AI systems that generate content that is plausible and coherent; however, the AI has not surfaced the reasoning for the content. Because of this, a clinician has no way to evaluate the reasoning for the record and the AI. The only element the clinician can evaluate is the content. This differentiates a record and the data from governance. More than most of governance conversations have acknowledged, this distinction matters. WHAT INVISIBLE RATIONALE ACTUALLY MEANS Take ambient documentation for example; a clinician sees a patient and AI creates a note and the clinician reviews it and signs off the note. The note reflects whatever was captured and processed by a model that the clinician did not create and cannot interrogate. If the note misses something, or mis frames a symptom so that it influences what happens next, the clinician who signed it has no rationale to review. There is output, but no reasoning. This is the case with AI-assisted triage and prioritization and clinical decision support where the recommendations are provided but no rationale is given for the reasoning. The impact on clinician reasoning is real, but the rationale is absent. When things go wrong and the chart is reviewed, accountability lands where it always has. With the person who attested to the decision. The difference now is that the rationale that would defend that choice is not available in any form to which they can refer. THE GOVERNANCE GAP It is not enough to ask whether an AI system is accurate. The harder question is whether a clinician can comprehend, review, and, if necessary, defend the rationale for what the system produced. If the answer is no, governance work is still incomplete, regardless of the state of the procurement process. WHAT THIS DEMANDS FROM HEALTH SYSTEM LEADERS Organizations that do this well are not slowing their adoption of AI. They are, however, changing the terms. For any AI-generated material to enter a clinical workflow to which a human will attest, there must be an explanation. No more black boxes with output. There needs to be an explicit reasoning process that the clinician can assess, challenge, and potentially replace. That requirement will revolutionize how you evaluate vendors. It alters what your governance committee wants to see regarding approvals to deploy that tech. It alters the support you provide to clinicians who will be using these technologies pretentiously. It also shifts the nature of the dialogue with your board. The question is no longer whether AI is performing responsibly and ethically in a clinical context. It becomes a question of whether those who are clinically responsible have the tools to defend their clinical decisions. FOUR QUESTIONS WORTH ASKING NOW 1. For each AI system creating content in clinical workflows, can the clinician access the rationale for what was produced, or just the output? 2. When a clinician co-signs a record influenced by AI-generated content, what review process exists? Is this review process realistic considering the time constraints? 3. Does your governance framework differentiate between AI that does the thinking for you and AI that just does the work? The attribution concerns are different. 4. If an adverse event were to happen tomorrow concerning an AI-generated clinical decision, would you be able to explain the rationale for what the system produced? The record has always belonged to the clinician. The least we owe them is a rationale they can actually see.

Keep reading