Building Ethical Intelligence in Healthcare

Related on The Signal Room: Healthcare Leadership and Operational Reality. Related from HDSC: Responsible AI in Healthcare.A healthcare system implements a predictive model to estimate risk for chronic disease progression. The model performs well on average. Six months later, an internal audit reveals model risk for underestimating people in lower income zip codes. The model training data is populated by people who have primary care bias. The problem is attributed to careless model monitoring after deployment. The initial model oversight consisted of a tick box and a checklist.

This is an ongoing issue. Healthcare system ethics reviews focus on pre-deployment and ignore post-deployment oversight. A tick box checklist is completed, a committee approves it, and the model goes into production. Little ongoing oversight is given to the model dynamic behavior in clinical and operational environments. The result is that ethical concerns only arise when they become visible problems, usually in the form of patient complaints, legal liability, or media coverage.

In 2021, the World Health Organization published a framework on ethics and the oversight of artificial intelligence in health and identified six core principles of AI in health: respect for autonomy, beneficence, justice, responsibility, explainability, and sustainability. These principles are often cited, but infrequently implemented.

Concurrently, the American Medical Association has instituted guidelines for the ethical employment of AI in medicine, including design requirements for AI tools aimed at eliminating inequities, coupled with the requirement for AI tools to be monitored continuously. AI is teaching physicians how to explain the technology to patients, but relatively few health systems have the supportive structures for such teaching to occur.

In The Lancet Digital Health, a publication analyzes gaps in the implementation of ethical principles at health systems. Most institutions found that ethical review of AI systems post-deployment lacked the staff, processes, and budgets to review the AI systems. Most ethics reviews, in the majority of cases, are a part of compliance, rather than a part of the organization.

Compliance asks whether a step in a process was completed. Ethics asks whether that step should be completed and whether its results are good. This is very important when discussing clinical AI, as a model may be completely compliant in the organization, and a model may still yield inequitable and clinically unreasonable outcomes.

Hastings Center has published findings regarding the healthcare technology field differentiation between rule-abiding and rule-bending. Compliance-focused reviews center around the minutia of data, rights, and regulations and miss the big picture involving inequities in the distribution of clinical impacts and the affected populations of the technology. Hastings Center, along with the National Academy of Medicine, has contributed to literature on the use of AI in health care and has identified the lack of proper regulatory mechanisms. AI technology, particularly its continuous adaptive algorithms and changing recommendations, poses ethical dilemmas that outdated regulatory mechanisms cannot handle.

In principle, ethical intelligence describes an ability to quickly identify, analyze, and act when ethical concerns arise regarding AI. This is not a committee that meets on a quarterly basis. Ethical intelligence is a collection of flexible and defined processes that enable leaders to respond to ethical concerns in a timely and efficient manner, immediately when an ethical concern is raised rather than weeks or months later.

Research from Stanford Human Centered AI Institute has spawned the study of organizational frameworks that enable efficient and responsible deployment of AI technologies. Institutions that effectively integrate ethical oversight into AI technologies have embedded ethical reviews into the operational structure of AI technology, rather than treating ethical oversight as a sequestered, independent layer functioning on a different timeline than the AI technologies.

Health Affairs published analyses on how health systems are beginning to invest in ethics infrastructure to match their investments in AI capability. The ethical investments argue that the expansion of AI in clinical and operational use cases necessitates a concomitant expansion of organizational capability to assess the ethical dimensions of these use cases, both prior to deployment and throughout the system lifecycle.

All of the components of ethical intelligence in a health system are definable. The first is a clearly articulated pre-deployment review process that includes clinical validation, fairness testing, and a population level impact assessment. The second is an enduring monitoring mechanism that assesses the model performance across different demographic and clinical sub-groups post-deployment. The third is an affirmative process that designates agency to modify, suppress, or retire a model when there are substantiated ethical concerns. The fourth is a concern-raising mechanism whereby clinicians, patients, and operational staff can bring forward their concerns, and are met with a timely and structured response.

ONC has issued guidelines to health IT developers and implementers that include the expectation of sustained monitoring and reporting. These guidelines pertain to AI systems, and are therefore useful for health system leaders as they build their internal structures.

Similarly, the IEEE has issued guidelines for the ethical design of autonomous and intelligent systems, including the health sector, and stresses the importance of sustained stakeholder engagement and visible decision-making frameworks.

For leaders in the health systems sector, the question is whether institutional ethical capacity is keeping up with the speed of AI and where the imbalances and gaps will increase over time. The institutions that will close this gap are the ones who treat ethical intelligence as an operational core function that will be resourced and measured with the same discipline as clinical, quality, financial, and performance metrics.

There are published WHO principles on AI and health ethics. The AMA has policy on the ethical use of AI in medicine. The Lancet Digital Health has published on the gap between ethical principles and practice. Hastings Center has published on the difference between compliance and ethics. The National Academy of Medicine has published on AI in health care. Responsible AI has been researched by Stanford HAI. Health Affairs has published on the need for ethics and AI. ONC has published health IT monitoring recommendations. The ethics of AI has been written about by the IEEE, as well as, standards for ethically engineered AI systems. This edition relates to the oversight and design in the themes outlined in Editions L, T, and V of this newsletter.

Christopher Hutchins
Founder & CEO, Hutchins Data Strategy Consultants