Only 3 of 1,357 FDA-cleared AI medical devices have ever been tested on whether they actually help patients

When a University of Toronto team set out to count how many of the FDA's cleared AI medical devices had actually been shown to help patients, they expected the evidence base to be thin. What they found was closer to none at all: of 1,357 AI and machine-learning devices the agency had cleared for use in patient care as of December 5, 2025, just three had ever been evaluated against whether patients actually lived longer, avoided a stroke, stayed out of the hospital, or reported a better quality of life.

The team, led by researcher Rawan Abulibdeh, built its device list from the FDA's own clearance database and the American College of Radiology's Data Science Institute catalogue, then cross-referenced every entry against ClinicalTrials.gov and PubMed to see what evidence, if any, backed each clearance. The results, published August 19, 2026 in *PLOS Digital Health*, run well past the one widely-quoted headline figure: of the 34 devices linked to a registered trial, only 12 (0.9%) had posted results on ClinicalTrials.gov, and separately, only 12 (0.9%) had a peer-reviewed publication to show for it -- the other 97.5% of the 1,357 devices rely entirely on retrospective or bench validation instead.

"Clinical decisions increasingly depend on algorithmic outputs. Yet despite this rapid adoption, one question remains largely unanswered: Do these tools actually improve patient outcomes?" -- Rawan Abulibdeh and colleagues, University of Toronto, in PLOS Digital Health

The gap traces back to how most of these devices reach the market in the first place. The dominant pathway, the 510(k) clearance process ("Cleared" and "approved" aren't the same word by accident. Most AI devices go through 510(k) clearance or the similar De Novo route; full PMA approval -- the tier that does require clinical data on safety and effectiveness -- is reserved for higher-risk devices and is comparatively rare in this category.), doesn't ask a manufacturer to prove a new device improves patient outcomes at all -- only that it is substantially equivalent to a device already being sold. A new AI tool can clear that bar by resembling an approved predicate, which can itself trace back to a predicate that was never outcome-tested either. Healio, citing the study, describes the effect as letting "evidence gaps propagate through chains of predicate devices, many lacking rigorous clinical validation."

Who gets left out of the validation studies that do exist is its own finding. The paper reports that imaging and cardiac-monitoring devices often excluded pregnant patients, adults over 75, and non-English speakers from whatever validation was performed -- populations the same tools may then be used on in practice, including in obstetric emergencies. Nearly three-quarters of the studies that did exist enrolled fewer than 500 participants.

The contrast with how the FDA treats a new drug is the plainest way to see what's unusual here. A new medicine ordinarily cannot reach the market without a trial demonstrating it actually helps the patients who take it -- that is the entire point of a Phase 3 trial. A 510(k) device clearance carries no equivalent requirement by design: the pathway was built in 1976 for things like updated forceps and improved catheters, where "basically the same as what's already approved" was a reasonable safety bar. Applying that same bar to software that reads a scan or drafts a diagnosis is the exact fit the researchers are questioning, not a flaw in how any single device was reviewed.

The researchers don't call for scrapping 510(k) review outright. Their stated recommendation is narrower and more workable: mandatory postmarket outcome registries for AI devices used in direct patient care, so evidence keeps accumulating after clearance rather than stopping the moment a device reaches the market. That would not slow initial clearance -- it would just mean someone keeps checking afterward, which for 1,354 of these 1,357 devices, nobody currently does in a way this audit could find.

The study's timing is what turns a bleak retrospective into a live policy question. One day before it published, on August 18, 2026, the FDA's own Digital Health Center of Excellence issued a discussion paper proposing how it might regulate the next category entirely: generative AI-enabled devices, including foundation models and agentic systems that go well beyond the pattern-matching tools this study audited. The agency is not proposing final rules -- it says so explicitly -- but it lays out a two-axis framework: how clinically significant the device's output is (informational versus diagnostic versus treatment-directing), crossed with how autonomous the system is (a clinician reviewing every suggestion versus a system that acts on its own). A chatbot that drafts a differential diagnosis for a doctor to accept or reject sits in a different box, under this framework, than an agentic system that orders a follow-up test itself.

The two documents are not the same finding wearing two hats, and treating them that way would overclaim what either one shows. The Toronto audit is retrospective -- it grades devices already on the market under a framework that predates generative AI entirely. The FDA's discussion paper is forward-looking and explicitly preliminary, aimed at systems that don't yet mostly exist as cleared products. What connects them is the question neither has answered: whether the next generation of AI devices, arguably more autonomous and less predictable than the pattern-matching tools this study covers, will be held to the outcome-evidence bar the last generation mostly skipped.

None of this means the 1,357 cleared devices are unsafe, or that a clinician using one is doing something wrong -- accuracy and bench testing are real evidence, just not the same evidence as a trial showing a patient outcome changed. 0.2% (of FDA-cleared AI medical devices have ever been evaluated on whether patients actually did better) What the study establishes is narrower and, for a reader trying to weigh a specific recommendation, more useful: "FDA-cleared" is a claim about resemblance to an existing device, not a claim about proven benefit, and for all but three of 1,357 tools, no one has yet checked the second thing at all.