The Patient Who Said No

A physician noted that advanced AI integrated predictive modeling identified the patient as having a considerable risk of a cardiac event in the following three months, and recommended more immediate workup to further evaluate. The patient then posed the first question in this sequence, "What if I do not want my data used in that model?" The physician was left in silence. No protocols exist for this situation. The patient signed a generalized consent form to register that does not specifically address the use of their data in AI. The EMR has no mechanism for AI opt-out. The clinical data flow has no alternative for this scenario.

The patient question was simple, common, and the institution had no clear plan. This is not an extreme situation. Health systems that lack the capability to address patient refusals for AI generated recommendations, or the data exclusion requests for AI systems, will be forced to address these issues ad hoc in clinical situations that require standardization and thorough documentation.

Patients keeping the right to refuse treatment is of great importance to medical law and ethics. Within the doctrine of informed consent, patients have the autonomy to refuse further actions after being informed of the risks, benefits, and alternatives. In fact, when a physician suggests a diagnostic test or a treatment, the patient has the right to decline, and the clinical team should document the refusal and act accordingly.

However, we cannot be sure if this extends to the recommendations of AI and the employment of patient data in AI models. The legal terrain is shifting. One of the main legal roadblocks is HIPAA, which protects patient information when it comes to treatment, payment, and health care operations. Many health systems have used this to justify the training of AI models and use them without full patient consent. In the U.S. and abroad, legislation like the CCPA and similar laws are placing restrictions to ensure individuals are granted various rights concerning the disposal of their information, including the right to not be subjected to an automated decision.

Regarding the intersection of AI and patient autonomy, the American Medical Association (AMA) has argued that patients should be able to know when they are receiving AI-assisted care and should be able to make decisions about the ways their information is processed. The AMA has positioned itself in favor of the notion that patient participation in the use of artificial intelligence should be informed and voluntary, however, this notion is in stark contrast to the current state of most health care systems.

The potential clinical and technical consequences become apparent once a patient refuses an AI-generated recommendation or chooses to exclude their data from a model. Clinically, a care team must find another way out. If the AI risk score is not available (or refused), how does a clinician make a risk determination, and what documentation is needed? Is the clinician at risk for a less favorable outcome that might have been avoided had the AI recommendation been followed?

Technically, the exclusion of one patient from an AI model is far more complicated than it seems. Most clinical AI models train on population data and make real-time predictions based on the data of the incoming patient. Along the prediction pathway, excluding a patient requires a technical solution that most EHR and AI systems do not currently have at a patient-specific level.

The Regenstrief Institute has researched the potential infrastructure to support patient-level data use controls across various health IT systems. Their research concluded that the systems currently available are optimized for managing data at the population level and do not have the necessary mechanisms available to exercise control at the model input level for individual patient preferences. The development of this level of control entails investing in data architecture, access control mechanisms, workflow interlocks, and other integrated systems.

Health systems that want to be prepared for the patient who says no need to examine three structural gaps. The first is a policy framework that clarifies the rights of patients regarding the use of their personal data by AI systems and AI-generated clinical recommendations. This framework should detail what patients can choose to avoid, how the opt-out preferences are documented, and how these preferences are communicated to the clinical and technical teams.

The second gap is a clinical workflow that allows for a refusal to be documented without creating excessive burden to the care team. Research published in the Journal of General Internal Medicine has studied the implications of a clinical decision support opt-out on physician workload and found that institutions with pre-defined alternative pathways experienced fewer disruptions and more consistent documentation than those who were managing refusals in an ad hoc manner.

Lastly, there is a need to develop the technical infrastructure that can implement patient preferences at the data pipeline level. The Office of the National Coordinator for Health IT has proposed patient data access and interoperability standards that can serve as a point of departure, although the AI-related data control gap still exists among the federal standards.

When patients deny consent for a procedure, a larger question arises regarding how AI technology has been integrated into care delivery models. The question extends past a single encounter. What is the authoritative framework that determines the extent to which patients can exercise their rights with respect to AI technologies integrated into the care delivery system? If patients can refuse a blood draw, deny a prescription, or abstain from participating in a research study, then, consistent with the values systems, patients should be able to refuse the AI-generated recommendations or the use of their data for AI purposes. Health system leaders, therefore, are faced with an urgent need to operationalize this principle within their policies and procedures, as well as technologies, instead of the principle remaining as an abstract concept that the institution is ill-equipped to pursue.

The right to refuse treatment is clearly defined within the informed consent doctrine. The use of PHI is governed by HIPAA, while many state privacy laws, most notably the CCPA, grant patients greater control over their data. The American Medical Association has addressed the intersection of AI and patient autonomy, while researchers from the Regenstrief Institute have explored patient-permission controls on the data used by health systems. The Journal of General Internal Medicine has studied how the ability to opt out of clinical decision support has impacted workflow, while the Office of the National Coordinator for Health Information Technology has established guidelines related to patient access to their data. This edition of the newsletter relates to patient rights and consent themes in editions AA, W, and Y.

Christopher Hutchins
Founder & CEO, Hutchins Data Strategy Consultants