When the Patient Brings the Algorithm

The patient comes to primary care with a consumer AI health application printed summary. This summary contains a risk assessment from wearable data, a diagnosis list with probabilities, and a treatment path recommendation. She now asks the doctor if the care plan can be done according to the AI. Her doctor has never seen this AI, has no information about the AI for a clinical verification and has no information about the AI to modify the clinical workflow. What started as a routine follow up has completely changed the nature of the consultation.

This is no longer a hypothetical situation. Patients arrive at clinical consultations with AI health assessment summaries because these applications have become consumer friendly. Many health systems are ignoring this challenge and clinicians are left alone to manage in the moment with no institutional backup.

In the past two years, consumer facing AI health tools have drastically increased. Mobile applications can analyze data from wearable devices, interpret lab results, and even create differentials and personalized health recommendations. A large number of direct consumer tools forgo the thorough testing applied to clinical grade AI systems.

A study featured in JMIR examined consumer AI symptom assessment tools which showed that diagnostic accuracy is not uniform across the board. Some tools reach performance levels that are similar to general triage, whilst others recommend actions that are not clinically advised. Because the quality of the AI suggestion may not be apparent from the output alone, this is especially problematic for clinicians in practice.

FDA recognizes this issue in its discussion paper regarding health consumer AI applications. In general, these tools lack a current legal definition in health technology that qualifies them as medical devices, hence circumventing the usual premarket review process. Consequently, clinicians cannot review the tools for their diagnostic accuracy and clinical safety, while patients use these AI tools without any formal evaluation.

When an AI recommendation is presented by the patient within a clinical scenario, numerous questions arise in the first instance. Clinicians have to determine whether they should even engage with the AI output and, in particular, how to record this AI output, including how they might be liable for a recommendation they either opted to follow or chose to disregard.

A Mayo Clinic study has explored how doctors feel about patient care recommendations made by AI. Doctors feel frustrated when they are asked to review an AI recommendation they did not create and cannot verify. Researchers noted three responses. Some doctors dismiss the AI output and do not discuss it with the patient. Others review the AI recommendation and follow protocol, providing the patient with a standard care plan rather than the AI approved care plan. Finally, a doctor may attempt to reconcile the AI recommendation with the clinical judgment. Each response has a documentation component and a liability component.

AMA has acknowledged the issues with consumer-facing AI within a clinical environment and suggested that health systems create protocols for clinician response to consumer AI recommendations. They emphasize that the clinician has ultimate clinical judgment and decision-making authority on the course of action regardless of recommendations made by external AI tools. Documentation should therefore reflect clinical judgment and rationale independent of the AI recommendation.

Documentation omissions tied to patient-initiated AI recommendations are largely unaddressed within the health care system. When a clinician is faced with an AI recommendation, the clinical record should contain clear, concise documentation about the rationale and explanation of the clinical plan, including whether the plan deviated from the AI recommendation. Most electronic health care systems do not provide the necessary structured documentation fields and most health care institutions do not have clear documentation practices and standards.

JAMIA has analyzed how physicians document patient encounters involving external health data, including AI content. Their analysis found that physicians used free-text notes with poor and unclear verbiage, leading to documentation that would be nearly impossible to justify in an audit or defend in court.

ONC has also recognized the gaps in standards for interoperability with patient health data, including data from AI-enabled consumer devices. Standards are defined in some areas of patient-generated data, and do not address situations where patients bring AI-generated clinical recommendations to the healthcare setting.

Health systems that want to prepare for the patient with the algorithm have three considerations. First, health systems have to prepare physicians to respond to externally generated AI outputs. This preparation includes how to document the encounter, how to explain clinical reasoning to the patient, and how to respond when the AI recommendation contradicts the clinical decision.

Second, health systems must provide educational resources that explain health consumer AI technologies to patients. This should not discourage patients from using consumer AI technologies, and health systems must ensure that patients understand the limitations and the clinical applicability of these tools.

Third, clinical leadership must track the consumer AI landscape to understand which tools patients in their populations are using and what those tools are recommending. Research from Stanford Medicine has examined health system strategies for monitoring consumer AI adoption among patient populations and found that early monitoring enables more effective clinician preparation and more consistent institutional response.

At its core, this is a relational issue. Patients who bring AI-generated health information into clinical encounters are expressing a form of engagement with their own care that health systems should view as constructive. The danger is that clinicians, under time pressure and without institutional support, respond in ways that damage the care relationship rather than strengthen it.

For health system leaders, the question is whether the institution has equipped clinicians to respond to informed, AI-enabled patients in a manner that is clinically sound, legally defensible, and respectful of patient participation. If the answer is no, that gap will widen as consumer AI tools become more capable and more prevalent.

Accuracy of consumer AI in assessing physical symptoms has been published in the Journal of Medical Internet Research. FDA has published a discussion paper about the status of consumer health AI. Research done by the Mayo Clinic has looked at clinician reactions to patient-requested AI recommendations. AMA has published consumer AI-related articles in a clinical environment. Documentation of health data outside the clinical environment has been described in JAMIA. ONC has stated that there is a need for interoperability standards for patient-generated health data. Stanford Medicine has researched the monitoring of consumer AI use by health systems. This edition is related to patient autonomy and consent topics covered in Editions AA, AC, and AF of this newsletter.

Christopher Hutchins
Founder & CEO, Hutchins Data Strategy Consultants