Clinical AI Safety Shifts From FDA Clearance to Workflow Testing

The argument in brief

An opinion piece published this week makes the case that hospitals are now deploying artificial intelligence widely enough to need a second safety test. The first test is whether the model works. The second is whether the care process built around it works. The piece starts with a year-long NHS evaluation, launched September 9, that covers 104 operating theatres, endoscopy suites and catheterisation laboratories across eight hospital sites. The programme aims to find bottlenecks and make better use of capacity across more than 20,000 procedures a year.11 The author notes that this is an operational tool, not a diagnostic one. What deserves copying is the method: the technology is assessed at several hospitals under one shared framework before early results are used to justify expansion.11

The central claim is easy to state and hard to put into practice. A benchmark tells you about a model. It tells you nothing about who responds to an alert, what other work gets delayed, or what happens when inputs are missing or the system goes down.11 Across a wide range of recent reporting, that view is no longer a minority position. It is quickly becoming the main question in healthcare AI.

The regulatory floor keeps rising, especially in radiology

The push for workflow-level testing comes as the number of authorized AI devices keeps climbing. Analysts tracking the FDA's AI-Enabled Medical Device List report that its September 4, 2026 update had 1,614 entries, 181 of them with 2026 decision dates.1 A separate directory, citing the agency, says the FDA had authorized more than 1,600 AI-enabled devices by September 2026, and that the agency itself calls the list non-comprehensive.3

Published counts vary depending on which snapshot they use. One analysis of the first-quarter 2026 data put the total at 1,524 authorizations, with 92 added in that quarter alone.5 Another put the figure at 1,451 at the end of 2025, after 295 authorizations that year.8 Some guides still give looser figures, such as "more than 1,000," and tell readers to verify them.2 These numbers don't contradict each other. They reflect a list that grows faster than most coverage updates, and the trend is the same in all of them.

Radiology's share is just as consistent. About three quarters of authorizations are imaging devices: 1,164 of 1,524 as of the March 31 snapshot, and 69 of the 92 first-quarter 2026 authorizations.5 Of the 181 entries dated 2026 in the September update, 135 list radiology as the lead panel.1 Large scanner makers hold the most clearances. Depending on the snapshot, GE HealthCare has between 120 and 130 radiology AI authorizations, ahead of Siemens Healthineers and Philips.15 Imaging AI is also moving toward broader, foundation-model tools. In January 2026, Aidoc reportedly received clearance for a body CT triage product that covers 14 conditions in one model.8

The framing in much of this coverage matters. One radiology-focused analysis points out that almost none of these products are "approved" in the strict sense. Most are cleared through the 510(k) pathway by showing substantial equivalence to an existing device.5 The op-ed draws the logical conclusion: marketing authorization shows that a device met premarket requirements for a defined intended use. It does not show that every hospital will get the same benefit, that every patient subgroup will be served equally well, or that local staffing can handle the work the tool creates.11 The author also notes that the FDA's January 2026 final guidance on clinical decision-support software clarified which functions fall outside the device definition. That is useful for manufacturers but answers a different question from the one a hospital board faces.11

What goes wrong between clearance and the bedside

The strongest evidence for the op-ed's position comes from documented failures in deployment. Guidance on post-deployment monitoring from the UK's Royal College of Radiologists includes an incident in which a radiographer saw an abnormality on a paediatric head CT but didn't escalate it, assuming the AI triage tool would prioritise the scan. The tool had not been validated for children. Adult and paediatric cases shared one reporting list, and staff didn't know the tool's limits.23 The department kept the tool for adults and changed its system so that every paediatric head CT goes to the top of the list.23 Nothing about the algorithm changed. The failure happened in the workflow.

The same document describes a mammography AI whose recall rate tripled after an imaging software update, flagging nearly half of studies until it was recalibrated for the new version.23 One industry review cites better-known cases: an external validation that found the Epic Sepsis Model had 33% sensitivity at recommended thresholds, and Google Health's diabetic retinopathy deployment in Thailand, where more than a fifth of images were rejected and screening slowed down.18

The evidence also points the other way. The op-ed cites a study published this month in npj Digital Medicine. In a neuroradiology department, combining an AI orchestration platform with structured change management raised the share of AI outputs that radiologists actively verified from 13.9% to 61.1%.11 The study was observational and did not show better patient outcomes, a limitation the author states openly.11 Still, it shows that implementation design can matter as much as model quality in deciding whether clinicians use a tool at all.

Monitoring is the weakest link

Reporting on what happens after go-live is the most worrying. A systematic review published October 15 in Frontiers in Digital Health found that 47.4% of hospitals did not evaluate accuracy after deployment and 51.4% did not assess bias. Where monitoring did happen, it was sporadic.28 The authors call for mandatory post-deployment monitoring with automated drift detection as part of a lifecycle framework for U.S. hospitals.28

Hospital leaders say much the same thing. Speaking to Becker's, Denver Health's Dr. Kortsch said the industry has spent years building rigor around validation before deployment and much less on what comes after. He added that ongoing monitoring may end up as the hospital's job rather than the vendor's.12 Stanford's Michelle Mello made a related point: the question health systems most often skip is whether a tool that worked in testing will work in the workflow it is deployed into, and whether anyone measures that afterward.15

The Royal College's guidance offers a simple structure for that measurement: people, process and product. In practice, that means checking whether clinicians use the AI as intended, whether it processes all relevant images on time, and whether its findings are accurate. The guidance also recommends that one named individual be accountable for monitoring.23

Fallbacks are the overlooked test

The op-ed's most distinctive argument is that switching an AI system off doesn't return a hospital to how it worked before. Roles, schedules and triage processes may have been rebuilt around the tool.11 U.S. health system executives have reached the same conclusion on their own. Cleveland Clinic's CIO told Becker's that once losing an AI capability can affect patient care or safety, it should be documented, tested and recoverable like any other mission-critical system.12 Cedars-Sinai is reportedly building an AI dependency map that tracks which workflows rely on which models, vendors and cloud services. Its leaders want tabletop exercises that include losing a model provider.12 Executives also raised a less obvious failure: a system that keeps running while quietly getting worse, such as a summarisation tool whose output degrades after a model change.12

The reading: hospitals now hold the burden of proof

The reporting largely agrees on the diagnosis. It differs mainly in emphasis. Consultancies and vendors frame the problem as governance and accountability. Accenture's health lead, for example, argues that AI influencing triage and treatment is clinical infrastructure and should be stress-tested as such.13 Clinicians at a recent summit in India stressed clinical validation in real settings.14 The op-ed narrows this into a specific checklist. Define the change in care you expect. Separate the model's effect from the effect of reorganising the service. Track subgroups and human factors. Rehearse outages. Set expansion criteria in advance.11

The overall picture is that the main constraint on clinical AI, and imaging AI in particular, has moved from regulators to hospitals. The FDA list keeps growing at about three quarters radiology.5 Yet the reporting suggests roughly half of hospitals don't check whether those tools still perform once installed.28 Regulatory clearance has become a minimum standard, and hospitals are now responsible for proving that a tool works in their own setting. The op-ed's central argument is that institutions should be able to show their care system performs better because of the model, not only that the model performs well.11 That is the right standard. The evidence so far suggests most hospitals are not yet set up to measure it.