The Oversight Debt Curve

Related on The Signal Room: Privacy and AI Governance Insights with Andre Samokish. Related from HDSC: Responsible AI in Healthcare.A healthcare system has implemented five AI-enabled clinical solutions in the past 18 months. Each tool is integrated with the clinical guidelines and training modules for end users after vendor assessments and approvals. No system is in place to track the AI tools currently in use, nor is there a way to determine their efficacy after they have been deployed. And no one individual is responsible for ensuring that the tools output results that are clinically appropriate, accurate, and consistent with the standard of care. The system has adopted AI technologies, and there is only a minimal structure to manage these tools. Tools are deployed in the healthcare system before their utility is established, thus creating a backlog that cannot easily be addressed. This is leading to an enormous debt that the organization will eventually be responsible for. The larger question is exactly how much this debt will cost when the organization is responsible for it.

Integration of the tools without oversight and documentation will eventually lead to an uncontrolled increase in frustration and dissatisfaction. Debt of this kind is inevitable, and the tools being deployed without oversight will only lead to greater frustration in the healthcare system. Deploying AI technologies without the necessary oversight is creating a backlog for the organization. This is the overwhelming reality. The real question is not whether the debt will exist, and the answer is that it will. What makes this most frustrating is that the deployed technologies without the necessary oversight will be the weakest part of the system.

Oversight debt in healthcare AI builds up in three areas. First, it builds up due to AI enterprise inventory. In Health Affairs, one article outlines how health systems are able to inventory AI systems in play, and most systems are unable to do so for their own organizations down to the operational and clinical floors. Without inventory, oversight has no starting point.

Second, oversight debt builds up in the gap that exists between deployment and monitoring. Brookings Institution has documented the post-deployment monitoring that health systems engage in when using clinical AI, and most lack a clearly defined evaluation of AI tool performance after the initial implementation period. Tools that were accurate at deployment may drift in performance as patient populations change or clinical protocols are updated. Without this monitoring, the drift that may cause the most harm will never be detected.

Third, oversight debt grows when there are no clear lines of institutional ownership. The American Hospital Association has researched how health systems assign responsibility for AI performance and found that there is no one role or committee designated to own AI tool deployment and performance across the enterprise. Responsibility is horizontally spread across leadership positions in Information Technology, clinical informatics, and quality.

The most dangerous part of oversight debt is that it compounds. Each new AI tool added to an environment lacking oversight does not add problems incrementally. Instead it exponentially increases the complexity of the problems because of how each tool interfaces with clinical workflows, documentation, and other AI tools, creating more dependencies and more failure points.

MIT has done research into the interaction effects of multiple unmonitored AI tools in the same clinical environment and has determined that the risk of negative outcomes increases disproportionately with the number of unmonitored AI tools that are present. An organization with two unmonitored AI tools has a more manageable risk than one with ten unmonitored AI tools.

In this regard, the Joint Commission has started to respond to the operational risks of unmanaged AI in healthcare. Their position states that healthcare systems need to consider the management of AI tools at a system level instead of at a unit level, and that patient safety incidents that could be avoided with managed processes and controls are more likely to occur because of the lack of central oversight.

The longer healthcare systems carry oversight debt, the costlier it becomes. Building oversight systems after deployment is operationally disruptive because of the tools already integrated into established clinical workflows. Clinicians are accustomed to using particular technology and AI processes, and institutional standards are set at the pace and productivity of AI-enabled operations.

Research conducted by KLAS has illustrated the operational disruption retroactive AI management initiatives present to health systems. Specifically, KLAS reports that those who constructed oversight systems post-deployment of several AI tools spent more time and resources compared to those that implemented oversight systems concurrently with the AI tools. Additionally, late adopters reported the most clinician pushback because the added processes of monitoring and reviewing were seen as yet another burden to already optimized workflows.

For healthcare leaders, the message is direct. Without investing in AI oversight systems, the operational debt for building the systems continues to accumulate with every passing quarter. There is no linearity in the debt curve. It accelerates, and the institutions that wait the longest will pay the most.

Due to the oversight debt, healthcare systems wishing to avoid the most disruptive outcomes must act quickly to implement three required capabilities.

To begin, a centralized AI inventory must retain information about the tools deployed, the areas of operation, clinical decisions impacted, and the individuals overseeing the systems. This inventory must be treated as a living document requiring updates with every new tool introduced or with every alteration made to existing tools.

Additionally, every health system must deploy AI systems accompanied by clear post-deployment monitoring along with a routine evaluation system. Monitoring must include drift mitigation strategies with metrics focused on accuracy, consistency, and clinical appropriateness, and must be designed to detect performance drift before resulting in injury.

Moreover, a health system must also specify an enterprise-level AI oversight owner. This does not constitute a new department. It places a defined role that spans across one or more of the clinical, operational, and technical domains to ensure adoption oversight.

Institutions that build these capabilities now will be positioned to scale their AI investments with confidence. Delaying implementation will add ever-increasing debt in the form of unmanaged AI that defines the limits of the organization.

Health Affairs has researched how health systems monitor AI integration. Brookings Institution has researched clinical AI post-deployment monitoring. AHA has studied how health systems assign responsibility for AI performance. MIT researched the interaction effects of several AI tools in clinical settings. Joint Commission has studied the operational risks of unmanaged AI proliferation. KLAS has studied the costs associated with retroactive AI management. This edition relates to the operational and institutional themes in Editions Z, AB, and AD of this newsletter.

Christopher Hutchins
Founder & CEO, Hutchins Data Strategy Consultants