The Most Dangerous Bias in Healthcare AI

A predictive model successfully identifies patients at high risk for complications post-discharge. The success of the predictive model points to a decrease in readmission rates. Leadership lauds the program as a successful deployment of AI technology.

Eighteen months later, a pattern emerges from an internal review. There is a common denominator for the patients the model overlooks. Patients utilize safety-net facilities. Their records contain gaps from periods of being uninsured. Their care is poorly documented in silos. The model was never trained to identify their data, because it was often absent from the training set.

The model was not elitist. It simply mirrored the data it was built on. The data mirrored a divide in the healthcare system, and the patients it failed to serve had been abandoned.

The most common form of algorithm bias involves well-articulated inaccuracies in the data with poorly defined variables and miscalibrated weights. This bias is identifiable and often correctable. The more insidious bias is an absence, and it often goes undetected because it does not appear as a miscalibrated error in the model. It is what the data lacks entirely.

In 2019 Obermeyer et al published a landmark study in Science that documented a pattern in a health algorithm that affected millions of patients. The algorithm considered healthcare spending an indicator of healthcare need. Due to structural inequities, Black patients with the same disease burden were spending less. The algorithm underestimated their need for care as a result. The bias was an inequitable spending assumption that was buried in the data.

Bias in data can be systemic, and Obermeyer et al have transformed the algorithm fairness debate. The study showed that bias does not require an intention. It merely requires that data carries the result of a biased system, and that the designers of the model do not appreciate the importance of what the data left out.

Healthcare data inequities reflect access inequities. Individuals who move between systems, experience coverage interruptions, seek care in under-resourced settings, or have a distrust of the healthcare system and avoid it altogether generate sparse, incomplete, or absent records. Predictive models trained on data that over-represents commercially insured populations in well documented health systems are optimized for those populations. Everyone else becomes noise.

In 2020, Vyas, Eisenstein, and Lyons wrote a review article that highlighted disparities in race-based adjustments in clinical algorithms spanning several clinical specialties. In their review, they described several race-based adjustments in clinical algorithms that resulted in underallocation of clinical services to Black patients. The consequences of their review reach past the clinical algorithms themselves and into how clinical data and clinical reasoning incorporate biased data to create tools that are presented as equitable to the end-user.

One of the goals of the NIH All of Us Research Program is to gain a better understanding of the data gaps and to create a research data set that reflects the true demographic, geographic, and socioeconomic diversity of the input patient population. The hypothesis of the program is that, as the evidence suggests, poor data quality at the point of construction results in poor data outputs, and no amount of data quality control on the back-end can compensate for the poor data quality in the front-end construction.

Bias from the exclusion of data requires an oversight system built to look for gaps, which is different from most model validation systems. In model validation, the primary metrics that are validated are accuracy, sensitivity, and specificity. There are no mechanisms to validate those metrics for disparities in different subpopulations across ethnicity, geography, payer type, language, or care continuity.

The American Medical Association recommends disaggregated performance analysis and model transparency and participatory oversight as part of their Health Equity Principles for AI. The NIST AI Risk Management Framework also considers demographic representativeness in its Risk Analysis Framework as one of the key dimensions of data quality. Both sets of recommendations converge on the same operational reality: for health systems to evaluate what they do, they need to measure it. Unfortunately, most health systems do not evaluate model performance by subpopulation consistently.

This is the intersection of oversight and operational reality. A health system that adopts a predictive model and that neither considers what data is missing from the model training datasets nor analyzes if model performance is worse for certain sub-populations has developed a tool that may perpetually worsen the socio-economic status of the very patients it is meant to help. The value at risk from this situation is not a hypothetical. It is being measured in real time at every health system that is hastily implementing clinical AI.

The solutions to data exclusion bias are well documented. These include oversampling the under represented, fairness-aware learning, stratified validation, and feedback loops that identify performance gaps. The barrier to closing those gaps is organizational priority.

Closing data gaps caused by systemic inequity starts with health systems confronting the social inequities directly. Health systems have data disconnected across care settings due to lack of investment in documenting the data and systems that support safety-net populations. The lack of data standards and interoperability across community health systems are the same issues this newsletter has covered in previous editions in terms of data integration, technical debt, and organizational maturity. The AI bias problem is a downstream impact of those primary issues.

Health systems that believe algorithmic fairness is a problem of model validation while leaving data infrastructure unchanged will find themselves in an endless cycle of model bias, model updates, and the same inequities in their next model iteration. The data is the problem, and bias is an indication of the inequity and lack of investment a health system has made in documenting and designing systems of care.

Obermeyer et al. published a landmark study in Science (2019) documenting how spending-based proxies systematically underestimated care needs for Black patients. Vyas, Eisenstein, and Lyons published a review reconsidering the use of race correction in clinical algorithms in The New England Journal of Medicine (2020). The NIH All of Us Research Program addresses data representativeness in biomedical research. The American Medical Association has published principles on equity in AI. The NIST AI Risk Management Framework identifies demographic representativeness as a core data quality dimension.

Christopher Hutchins
Founder & CEO, Hutchins Data Strategy Consultants