The Tools You Did Not Approve

Was this edition forwarded to you?
AI Health Pulse is a weekly briefing on healthcare AI strategy and oversight, written for health system leaders. Get every edition in your inbox.
Subscribe at https://aihealthpulse.beehiiv.com/subscribe

In a 2024 study by BMJ Health and Care Informatics of about one thousand UK general practitioners, one in five reported utilizing generative AI applications (including ChatGPT) for drafting clinical notes and for informal assistance with differential diagnosis. A critical caveat is that with the very few exceptions, most of the generative AI applications have not undergone any formal review by any of the controlling authorities. This is not intended, as many would hope, to be a prediction of what clinicians in the UK (and abroad) are thinking of doing in the near future, as the survey respondents are the clinicians that are most likely to be doing it. The survey is British, but the described pressures are not. If a system thinks their clinicians are behaving differently, it has confused a lack of measurement with a lack of use.

Almost all healthcare systems will be experiencing the same (as evidenced by this survey) or a very similar situation (as will be evidenced). The tools are all easily accessible through a web browser. They bypass all the procurement gates. No committee will ever be published on for a review enabling clinicians to use the tool(s). They use a documentation aid that saves at least twenty minutes at the end of a long shift. Any organization that thinks otherwise is working on a very different set of assumptions to those based on the reality of a clinicians work.

The Failure of the Ban

The typical first step is to send an all-staff email about the tools’ ban, and document the warning on the intranet, like has been done with past technologies. Such bans have been ineffective in controlling the use of personal phones, messaging apps, or thumb drives, and for the same reasons will be ineffective in this case. A ban does not solve the root cause of demand, and instead, gives the use of the technology a layer of concealment. People will continue using the technology on personal accounts in the most secure ways, and the people who know the most about the situation will have the most reasons to be quiet. The organization will be just as exposed as before, if not more, because now it has less intelligence on the situation.

There is also a fairness concern that will remain unaddressed by this ban, that it is upon the leadership of the organization to address. The organization created the situation that required the use of the technology to circumvent the documented burden, and then sent out the warning email about the shortcut. Clinicians know this already. A response that addresses and explains the demand will generate a lot of useable intelligence, where a response that threatens will result in silence.

That demand is certainly the signal that you should focus on. These tools are used because, to these clinicians, something was broken enough for them that the usage of an unvetted chatbot was an appealing option. I covered that side of the problem in my previous article “Issue 31: Shadow AI: The Symptom, Not the Threat.” This article will focus on the challenges in the response. Understanding the demand is important, but it does not answer the operational problem of what you should do with the tools available to you come Monday.

The Formulary Move

I spoke about many topics with emergency physician Dr. Ömer Atli on The Signal Room. Dr. Atli stated that the most difficult instances of AI safety challenges are where medicine is the thinnest, for example in under-resourced and rural areas. He also gave me a framing device which has remained with me. He referred to the unregulated use of AI within a clinical context as the shadow formulary. For the past century, hospitals have constructed a functional means to manage the presence of novel, powerful drugs that emerge faster than the ability to prove their safety and efficacy—this is the construct of the formulary and the committee. They review evidence, conditionally approve a drug, and monitor it post-implementation to withdraw it if the evidence changes. No one argues against the use of pharmaceuticals writ large. The institution decides what belongs on the list, and the list is maintained by those who are assigned the duty of maintaining it. Listening to him build the case for a formulary is worth an executive hour: https://signalroompodcast.com/episodes/clinical-ai-shadow-formulary

When it comes to AI, the inversion of the ban is applicable. The first step is recognizing that the shadow inventory is already established, and then norms will be instituted to codify it.

The example of conditional admission pairs well with a tool. A tool granted conditional admission leads to the granting of an institutional account. This means that use is monitored and the data terms are the ones the institution negotiated, including an agreed-upon written set of tasks that includes a clear statement of prohibited tasks and a date for review. A tool that was granted conditional admission in August is not safe in February. This model of conditional admission is an example of restricted access, which has been a practice with pharmacy for many years. Clinicians use this model every day and do not see it as a breach of trust.

As part of the building of trust paradigm, Conditional Admission requires honest inventories. This means that there are no consequences for answering the inventory, and if clinicians believe they will not suffer consequences for telling the truth, then the inventory will no longer be fictitious. The Data Inventory includes tools that clinicians use that collect patient data, as well as data that leave the premises, if any. These two tools will create a more valuable inventory for the organization than any written policy, because policies written without these tools will draw a fictitious picture of the organization.

What does this process look like? Don't think about audits. Think about a listening tour. It starts with an anonymous survey. Making the survey anonymous is the best way to guarantee the most honest feedback and first replies. This is followed by a series of department meetings. In these meetings, the promise is that nothing discussed will go into any sort of personnel or disciplinary file. When a tool is brought up, we want to know what task it performs and what data it works with, as well as how frequently the tool is used. If the tool is load bearing, or used as a part of daily work, it is not going to be removed because the work above the tool has adjusted itself to accommodate the tool and the time it provides.

The survey feedback we received is better than the hypothetical, and, thus, almost deserves its own section. Free consumer chatbots that save everything typed and clinicians who paste discharge summaries into them have formal built out data reviews. The fidelity to the privacy and data protection built around the ease-of-use of a tool is a far cry from the data protection commitments made to the patients, and this is a part of the inventory that cannot wait for a second pass. It defines the boundaries.

Beyond this point, logic applies. Some tools require a permitted account, data border, provisional acceptance, and use case, while others are substituted with a sanctioned equivalent that meets the same need and that’s the only actual intervention that deals with the demand. Others are simply removed with the justification explicitly stated. A review that admits some tools is one that most clinicians trust more than a review that only states the tools are not permitted.

Authorized alternatives are the only options that address the demand, while all other options are simply providing a cover. There are enterprise versions of the same tools with data terms that an institution can negotiate, as well as clinically focused tools that can substitute for what clinicians are attempting to achieve with the tools. The winning answer is the one that gets there first. If the authorized tool is going to add an extra three clicks with an extra login delay, the shadow tool is going to be used, no matter what the policy says.

This refers to data strategy work, and this is where these engagements begin. At Hutchins Data Strategy Consultants, we have found that the projects which originate from an AI question usually lead to this one. This occurs in the absence of any draft policy. If your system requires this level of comprehension, we will gladly assist you: https://hutchinsdatastrategy.com/services/ai-and-data-readiness

The Real Option

Management cannot select the option of whether to allow unauthorized AI to operate within the organization. That option was decided long ago by the app store and the corporate environment, long before any policy meeting was convened. The real option is whether to address a version of the problem that is visible to the organization, or a version of the problem that is not. A health system that builds an inventory, sets a review, and creates allowable options is not being liberal. Rather, it has traced the steps of its own pharmacy which began with a greatly improved health and safety perspective and with a more accurate inventory.

Context and Sources

This edition draws on the 2024 BMJ Health and Care Informatics survey of UK general practitioners on generative AI use in clinical practice and on the conversation with Dr. Ömer Atlı on The Signal Room. It continues themes from issue 31, Shadow AI: The Symptom, Not the Threat, issue 52, When the Software Stops Asking, and issue 45, The Procurement Trap.

The AI Health Pulse is a weekly briefing on healthcare AI strategy and oversight that provides independent insights for busy executives. It is written by Christopher Hutchins, a former health system data executive and the founder of Hutchins Data Strategy Consultants. The publication is free from sponsorship and paid placements. Each edition is based on named sources, and the operational aspects of healthcare AI strategy and oversight are covered, such as data readiness, model oversight, and the impact of AI on an organization. You can subscribe at https://aihealthpulse.beehiiv.com/subscribe. The complete archive is also free to access.

Companion reading: Best Healthcare AI Newsletters for Executives, An Honest List: https://aihealthpulse.beehiiv.com/p/best-healthcare-ai-newsletters-for-executives-an-honest-list

Christopher Hutchins
Founder & CEO, Hutchins Data Strategy Consultants

Continue reading from Hutchins Data Strategy

Healthcare AI Consulting: https://hutchinsdatastrategy.com/insight/healthcare-ai-consulting
Healthcare Data Strategy: https://hutchinsdatastrategy.com/insight/healthcare-data-strategy
Why Healthcare AI Fails at the Data Layer: https://hutchinsdatastrategy.com/insight/ai-trust-at-the-data-layer

On the Signal Room podcast

The Clinical AI Shadow Formulary: https://signalroompodcast.com/episodes/clinical-ai-shadow-formulary
AI Agents and Healthcare Security Risk: https://signalroompodcast.com/episodes/ai-agents-healthcare-security-risk
Balancing Human Judgment and Clinical Trust: https://signalroompodcast.com/episodes/balancing-human-judgment-clinical-trust

The Tools You Did Not Approve

The Failure of the Ban

The Formulary Move

The Real Option

Context and Sources

Recommended for you

Quick Links

Subscription

Socials