Accountability in Practice

Why External Forces Are Reshaping Accountability. In my work with health systems and analytics teams, I have watched accountability for AI evolve from a theoretical concern to an operational imperative. The shift has been gradual, but the acceleration is unmistakable. Several forces are converging simultaneously. Regulatory momentum is building faster than most realize. The FDA has now authorized over 1,250 AI-enabled medical devices. State legislatures introduced more than 250 AI-related health bills in 2024 alone, and thirty-three of those bills were passed and enacted into law across twenty-one states. California now requires disclosure when generative AI creates patient communications (AB 3030). Colorado mandates risk assessments and anti-discrimination protections for high-risk AI systems (SB 205). Illinois prohibits AI from making independent therapeutic decisions without licensed professional review (SB 3115). Texas requires written disclosure to patients before AI is used in connection with their care (HB 2060). This is not scattered activity. It is a coordinated tightening across jurisdictions. Payer scrutiny is intensifying. Medicare Advantage plans are facing lawsuits over algorithmic claim denials. CMS has clarified that AI cannot supplant physician judgment in medical necessity determinations. State after state is introducing legislation requiring human review of AI-driven coverage decisions. In 2025, approximately 60 bills were introduced aimed at regulating AI use by insurers, managed care plans, or other payers. When payers face accountability requirements, those requirements flow downstream. Providers using AI tools for utilization management, documentation, or clinical decision support will need to demonstrate the same transparency and oversight that regulators are now demanding from insurers. Legal exposure is expanding. The Department of Justice updated its Evaluation of Corporate Compliance Programs guidance in 2024 to specifically address AI misuse. Prosecutors are now instructed to evaluate whether companies are 'vulnerable to criminal schemes enabled by new technology, such as false approvals and documentation generated by AI' and whether 'compliance controls and tools are in place to identify and mitigate those risks.' The Texas Attorney General secured the first settlement against a healthcare AI company for misrepresenting accuracy metrics. The company had claimed a 'critical hallucination rate' of less than 0.001 percent. The state determined those claims violated the Texas Deceptive Trade Practices Act. Malpractice frameworks are evolving in parallel. If an AI tool contributes to a diagnostic error or treatment decision, the question of who is accountable, and what oversight was in place, will be central to any legal proceeding. Patient and public awareness is growing. Stories of algorithmic bias, chatbot failures in mental health contexts, and opaque decision-making are reaching mainstream audiences. In September 2025, the FTC launched an enforcement inquiry into AI chatbots acting as companions, with particular attention to impacts on children and teenagers. Trust is not automatic. Organizations that cannot clearly explain how AI is used in patient care, who oversees it, and how concerns are addressed will face questions they are not prepared to answer. What Accountability Looks Like When It Is Required. When accountability shifts from voluntary to mandatory, the structure changes. It becomes less about internal best practices and more about demonstrable, auditable processes. Based on the emerging regulatory landscape, several elements are becoming baseline expectations. Ownership that is visible and traceable. Every AI tool in clinical or operational use needs a clear owner. Not buried in a policy document, but visible to the people who use the tool and the regulators who may audit it. Ownership includes responsibility for monitoring performance, responding to concerns, and ensuring the tool remains aligned with its intended use. Decision pathways that are documented. When an AI system influences a clinical or coverage decision, there must be a clear record of how that influence was incorporated and who had final authority. The days of algorithmic recommendations flowing directly into action without human review are ending in many jurisdictions. Transparency with patients. Multiple states now require disclosure when AI is used in patient communications or care decisions. Indiana's HB 1620 requires healthcare providers and insurers to tell patients when AI is used in decisions or communication. This is not about lengthy consent forms. It is about clear, accessible information that allows patients to understand when AI is part of their care and how to raise questions if they have concerns. Regular review tied to real-world performance. Static validation at deployment is no longer sufficient. The FDA issued a Request for Public Comment in September 2025 specifically seeking feedback on 'evidence-based methods to identify and address performance changes over time' for AI-enabled medical devices. Regulators and accreditation bodies are increasingly focused on ongoing monitoring: how organizations detect drift, respond to performance changes, and ensure AI remains aligned with clinical reality over time. Audit readiness. California's SB 1120 requires health plans to make their AI programs and tools available for government audit and inspection. Similar requirements are emerging elsewhere. Organizations need to be able to demonstrate not just that they have governance policies, but that those policies are functioning. Why Accountability Breaks Down Even in Prepared Organizations. In most cases, accountability does not fail because of negligence. It fails because healthcare is complex and AI touches many domains simultaneously. A tool may be deployed by IT, configured by analytics, used by clinicians, and overseen by compliance. Each group sees part of the picture. No one sees all of it. The most common breakdowns I observe: A question gets raised but no one follows up because ownership is unclear. A change is made to a workflow but the AI tool is not updated to reflect it. A model is still 'owned' by a team that has since been reorganized or dissolved. A frontline concern does not reach the group responsible for addressing it. A metric shows drift but the next step is not defined. These are not dramatic failures. They are quiet gaps that compound over time. And in an environment where regulators, legal teams, and patients are paying closer attention, those gaps become visible faster than they once did. How Teams Build Accountability That Holds. Across settings, I have seen a few consistent practices that make accountability durable rather than performative. Make ownership findable, not just documented. The test is not whether ownership is written down somewhere. It is whether a clinician with a concern, an analyst with a question, or an auditor with a request can find the right person within minutes. If ownership requires excavating policy documents or navigating organizational charts, it is not functional. Create a clear front door for concerns. Strong accountability does not require complex governance committees for every issue. It requires a simple, trusted pathway for raising questions. When people know where to send a concern and trust that it will be acknowledged, small signals surface before they become large problems. Build regular touchpoints into the rhythm of work. Accountability works best as a steady cadence, not a reactive process. Short, recurring check-ins between clinical, operational, analytic, and technical partners keep alignment steady without requiring heavy process. The goal is to catch drift early, not to discover it during an audit. Treat transparency as operational, not promotional. Transparency requirements are not about marketing. They are about building the documentation and communication practices that allow patients, regulators, and internal teams to understand how AI is being used. Organizations that treat transparency as a compliance checkbox will struggle when the questions become specific. Prepare for the audit before the audit. The organizations best positioned for the emerging regulatory environment are those that can produce clear documentation of their AI governance on short notice. This includes inventories of AI tools in use, ownership records, performance monitoring data, and evidence of human oversight in decision pathways. Why Accountability Breaks Down Even in Prepared Organizations. The window for voluntary governance is narrowing. External forces are not waiting for organizations to be ready. In December 2025, President Trump issued an Executive Order directing federal agencies to challenge state AI laws viewed as overly burdensome, while calling for development of a 'minimally burdensome' national standard. Regardless of how federal preemption debates resolve, the direction is clear: accountability frameworks are coming, and they will be imposed rather than suggested. Healthcare AI is progressing faster than the governance structures designed to support it. That gap is closing, not because organizations are catching up, but because regulators, courts, payers, and patients are accelerating their expectations. Self-regulatory bodies are moving as well. In September 2025, the Utilization Review Accreditation Commission (URAC) released new accreditation tracks specifically for AI in healthcare settings. The Joint Commission, in partnership with the Coalition for Health AI, published guidance on responsible AI use and announced plans for a formal certification program. These are not distant possibilities. They are active developments shaping the standards organizations will be measured against. Keeping Accountability Grounded. Accountability is not a burden to be managed. It is the infrastructure that makes AI trustworthy, sustainable, and defensible. The organizations that will lead in healthcare AI are not those deploying the most tools. They are the ones building the governance structures that external forces will soon require of everyone. They are treating accountability as a capability to develop, not a compliance exercise to endure. In a field as consequential as healthcare, the question is no longer whether accountability will be demanded. It is whether your organization will be ready when it is.