The AI Readiness Diagnostic

A community hospital with 240 beds goes live on an AI imaging tool that flags suspected pulmonary embolism on CT chest. The tool routes positive flags directly to the radiologist queue with a priority tag. On day three, a radiologist marks a false positive that delayed a different read by forty minutes and asks operations who has the authority to pause the tool while the case is reviewed. Operations does not know. The radiologist asks the CMIO. CMIO escalation goes to the CIO, who confirms the integration was scoped and signed under his department but says the clinical halt decision belongs to clinical leadership. The clinical chief returns the question to the radiologist. Six hours later the tool is still running, the radiologist has stopped using its outputs, and the question of who can halt the tool has not been answered.

The question the radiologist asked was the first of five the leadership team would learn to ask before the next go-live.

The five questions surfaced in the post-deployment review on Monday. Four came from the readiness gaps the leadership team had not anticipated, and one came from the radiologist who asked first. Across most health systems by 2026, the infrastructure is in place, the clinical decision support tools are live and accumulating data, the data warehouse can serve the model training that vendors are promising. None of those measures individually tells the leadership team what it needs to know, which is whether the system can deploy the next AI tool without producing the same scene six hours after go-live. The integrating diagnostic that produces that answer lives at the intersection of the CIO, the CMIO, and the Chief Data Officer, in the seat the system has not yet hired.

Coalition for Health AI Blueprint v1.0 placed organizational readiness as the precondition for trustworthy AI deployment, naming it before model selection, before pre-deployment evaluation, and before lifecycle monitoring. AHA Trustees Toolkit on AI in health care framed the readiness question as the central board agenda item for 2024, and the answer shaped every downstream decision the board made about AI investment that year. NIST AI Risk Management Framework 1.0 puts organizational readiness inside its Govern function, with specific reference to senior leadership scoping, role definitions, and escalation paths as the structural elements that have to exist before the rest of the framework can do anything. The diagnostic exists in the literature. Most health systems have not yet operationalized it.

What boards get in place of a diagnostic is a survey. A consulting firm produces a sixty-page report that grades the system on a maturity scale and recommends additional investments. The report arrives between budget cycles, at the level of generality that makes it hard to act on, and by the time the next assessment runs the recommendations have aged out and the underlying conditions have changed. What works in place of the survey is a small number of operational questions the senior team can answer in plain language, on demand, with the same answer holding from one quarter to the next.

The first thing to ask is whether the system can stop a live model in twenty-four hours, and the answer gets at authority, procedure, and political reality at the same time. Most leadership teams find out the answer when they ask the question out loud for the first time, when the answer surfaces a chain of approvals that runs through legal, IT operations, contract management, and the vendor relationship team, with no single role authorized to pull the trigger alone. ECRI Institute named this kind of inability as one of the conditions that put AI at the top of the 2025 hazard list. The second thing to ask is whether the system has validation data that is independent of what the vendor used to train the model. Without local held-out data stratified by the equity dimensions the board monitors, the system cannot tell whether the model is performing for the patients it serves. AHRQ patient safety publications argued for this separation as a baseline condition for AI in clinical use, and ONC HTI-1 final rule requires disclosure on training data sources specifically so that local validation becomes possible. The third thing to ask is whether each deployed model has a named senior owner. Committee ownership turns out to be no ownership when something goes wrong. Vendor ownership is outsourcing. CIO ownership categorizes a clinical model as infrastructure, which most clinical models are not. Coalition for Health AI Blueprint v1.0 names the absence of senior individual ownership as the most common reason AI committees fail to develop operational authority in their first year, and Robert J. Margolis Institute work on AI committee composition shows the committees that survive year one have moved from collective to named ownership before month nine.

Fourth, whether the board sees model performance on a quarterly cadence, set up before incidents force the cadence into existence. When reporting starts only after a near-miss event, there is no way to distinguish a one-time event from a trend, and no way to direct resources before harm accumulates. The reporting view works on a fixed cadence with the same metrics quarter over quarter, presented by the same person each time. AHA Trustees Toolkit on AI named this reporting structure as one of the four scoping decisions that determine whether the senior AI seat operates with standing or without it. The fifth question, and the one most often skipped, is whether there is a workforce upskilling budget separated from the AI program budget. That line item is the test of whether the system is treating AI as a tool to deploy or as a change to absorb. Budgets that have not been carved out and protected do not exist in the operational reality of a health system. Duke-Margolis Institute work on AI committee composition treats workforce upskilling as a separately funded line item in the committees that have produced operational AI, and the absence of that line is one of the cleanest signals that a system has overestimated its readiness.

The diagnostic produces a yes-or-no for each of the five, and the system either has the capability or does not. Systems answering yes on all five are the ones that have already operationalized at least one mature AI use case under named senior leadership. The next tier, with yes on three or four, are the systems most likely to be hiring the Chief AI Officer seat in 2026 and reading the diagnostic as a planning instrument. With two yes counts or fewer, the next AI investment hits the same readiness gaps the prior investment did. Budget keeps growing. The trajectory does not change.

What happens at the lower end is a known sequence. The model gets deployed because the vendor presented a compelling case and the board wanted to be on the forward edge. Six to nine months later, an incident occurs. The incident exposes the same five gaps every time: the model cannot be halted on demand, no local validation data exists, ownership was never assigned, the board never started seeing performance reports, and upskilling for the workforce did not get funded. Each of those gaps could have been closed if it had been the only one. The board has to address all five at once after the harm. Recovery takes two years.

ECRI Institute documented this pattern when it ranked AI as the top health technology hazard for 2025. The Joint Commission RCA2 methodology has been the standard tool for analyzing safety events since 2015, and it was built for causes that can be separated into human, technological, and process layers. AI events do not separate that way.

In the room are the CIO, the CMIO, the Chief Data Officer, the CFO, and the senior AI leader if the seat exists yet. An hour is what the meeting takes. The five questions go in order. Answers come back in writing during the meeting. The document goes to the board at the next quarterly review. Writing the answer down forces decisions the survey would have kept abstract. Once the system has to write down it cannot halt a model in twenty-four hours, the team either fixes that condition before the next quarter or carries it to the board with the gap visible.

A baseline comes out of the first conversation. Each quarter the same five questions get asked, with the same documentation discipline, and the trajectory of yes-counts over four quarters becomes the readiness vector the board can use to direct investment. Systems moving from two to three to four to five yes-counts over a year are systems on a credible AI trajectory. Stuck at two over the same year, and the next AI investment amplifies a gap the investment cannot close on its own.

Coalition for Health AI Blueprint v1.0 names this kind of disciplined readiness work as the prerequisite for everything else the framework requires, with specific reference to the fact that readiness assessments occurring once age out faster than the underlying conditions change. The frameworks adopted at health systems running mature AI portfolios share this pattern: readiness is treated as a standing diagnostic, with the diagnostic owned by the senior AI leader and reviewed by the board on a fixed schedule.

The five questions that surfaced after the community hospital go-live are the same five questions that surface in every readiness conversation worth having. A system that can answer them in plain-language yes-or-no responses is a system that has done the institutional work the next decade of AI deployment will require. ECRI Institute keeps elevating AI on its hazard list because the readiness gap is the structural condition producing the harm. The diagnostic costs nothing to run. It takes one hour, five questions, and a senior team that puts the answers in writing.

This edition draws on positions from the Coalition for Health AI Blueprint v1.0, the American Hospital Association 2024 Trustees Toolkit on AI in health care, NIST AI Risk Management Framework 1.0, ECRI Institute Top 10 Health Technology Hazards reports for 2024 and 2025, AHRQ patient safety publications on AI validation and lifecycle monitoring, the ONC HTI-1 final rule on training data disclosure, the Joint Commission RCA2 methodology applied to AI events, and the Robert J. Margolis Institute for Health Policy at Duke University 2024 publication on AI committee composition. Related editions: Issue 39 (Your Board Will Ask About AI. The Question Will Come Too Late.), Issue 41 (Where Responsibility Breaks Down), Issue 42 (The Innovation Tax), and Issue 43 (The Chief AI Officer Mandate).

Christopher Hutchins
Founder & CEO, Hutchins Data Strategy Consultants