AI Medical Diagnosis Study: Microsoft MAI-DxO Hits 83% Accuracy

A benchmark win, published in Science

Microsoft has published its strongest evidence so far that AI can reason through a diagnosis instead of just picking an answer from a list. A peer-reviewed paper in Science, titled "Sequential Diagnosis with Language Models," reports that the company's Microsoft AI Diagnostic Orchestrator, or MAI-DxO, correctly diagnosed 83.2% of a set of very difficult case records from the New England Journal of Medicine. Generalist physicians allowed to use textbooks and the internet got 36.1% right.48 Doctors working without references averaged about 22%. That gap is the source of Microsoft's "nearly four times as accurate" claim, while the "more than twice as accurate" claim compares the system with doctors who had references.29

The paper also adds a second, more ordinary test set: emergency department admissions drawn from the MIMIC-IV-ED dataset, which comes from Beth Israel Deaconess Medical Center in Boston. On those cases the system scored 80.3% and physicians scored 46.8%, which Microsoft describes as 1.7 times better.42 Microsoft AI chief Mustafa Suleyman calls the results a step toward "medical superintelligence."6

My read is that the headline numbers are real but measure something narrower than they suggest. The study shows that a carefully built system on top of frontier language models does very well in a controlled, text-only simulation. It does not show that patients get better care. The next question for the field is how a tool like this gets regulated and deployed.

How the test worked

The method is the most interesting part of the work. Microsoft built a benchmark called SDBench. It turns 304 NEJM clinicopathological conference cases and 325 emergency-department-to-admission encounters into step-by-step diagnostic exercises.8 Each case starts with a short vignette. The diagnostician, human or AI, can then ask history questions, request exam findings, order tests or commit to a diagnosis. A separate AI "gatekeeper" reveals information only when asked, and every test carries an estimated price based on hospital chargemaster rates.10

This design answers a fair criticism of earlier medical AI claims. Microsoft's Dominic King noted that models now reach near-perfect scores on multiple-choice exams, which look nothing like real practice. His summary: "No patient ever came to see me with a nice summary."10 Inside the system, several AI agents act like a panel of doctors with different roles. They debate hypotheses and weigh whether each test is worth its cost.610

The system also worked differently from the doctors. Physicians asked an average of 5.6 questions before ordering their first test, while the orchestrator running on GPT-5 asked 25.5.8 When the AI was wrong, it usually named a closely related diagnosis at the wrong level of detail. Physician errors tended to be further off.10

Where the coverage diverges

Reports of the study differ in ways that matter. Some coverage leads with the 2025 preprint figures: 85.5% accuracy against a 20% physician average, which made the gap look more than fourfold.1 The peer-reviewed version puts the headline accuracy in the low 80s and pairs it with a higher physician baseline when references are allowed.36 Accounts that mix the two versions make the result look bigger than the published paper supports.

The cost claim is the clearest split. Reuters-based reporting says workups designed by the orchestrator were cheaper than those ordered by doctors.10 A closer reading of the paper complicates that. With no budget limits, the AI's average estimated testing cost per case was higher than the physicians': about $5,431 versus $4,725 on NEJM cases, and $4,979 versus $3,400 on emergency cases.910 Savings appear only when budget limits are imposed. One setting reached 80.9% accuracy at $2,241 per case and another 79.6% at $1,303.8 Those figures still beat the physicians, but they give up some accuracy, and all of them are estimates from standardized U.S. pricing, not measured hospital spending.10

The second point of disagreement is who deserves the credit. Several analyses point out that much of the improvement comes from stronger underlying models, mainly OpenAI's GPT-5 and GPT-5.5, rather than Microsoft's orchestration layer.37 GPT-5 with no orchestration scored 76.6% on one comparison. Adding MAI-DxO raised that by 3.7 percentage points and added roughly $740 per case in estimated cost.9 Averaged across models, orchestration added 7.9 points on the NEJM cases and 4.4 on the emergency cases, with the biggest gains for weaker models.83 One newsletter put it bluntly: how much of what is branded "medical superintelligence" is just well-prompted OpenAI?7 That framing is a little unfair. A model-agnostic harness that improves whichever model sits underneath has value, and Microsoft argues it removes the need to chase each new model release.10 Still, the gain from the harness is much smaller than the gain over the doctors in the study.

What the study leaves out

The authors acknowledge important limits. The human comparison group was individual generalists, not specialists or multidisciplinary teams, who normally handle cases this hard. The authors say they expect the gap to narrow against specialists.45 The NEJM cases are picked for teaching value and all had a confirmed pathological diagnosis, so healthy patients and benign visits were excluded.10 Physicians also had to work alone, without colleagues.1

There are also questions about how the data were split. The doctors with references were tested on a 56-case held-out set, while the AI's headline accuracy covers the full 304-case collection.9 The best single physician scored 51.8% on NEJM cases and 62.5% on emergency cases, which suggests the spread among humans is wide.8

Most important, the benchmark is text only. It includes no imaging, waveforms or pathology slides, and it leaves out the messiness of real encounters, such as patients who forget or leave out details.10 It measured diagnosis only, not treatment decisions, safety or outcomes.9 There is a relevant warning from earlier research: a 2024 randomized trial in JAMA Network Open found that giving physicians a chatbot did not significantly improve their diagnostic performance. A model scoring well on its own is a different thing from a doctor-and-AI team doing better.9

The radiology gap

The text-only limit stands out given where clinical AI already operates. The FDA's AI-Enabled Medical Device List, updated September 4, 2026, includes 1,614 authorized devices with decisions through June 29, 2026. Radiology accounts for 1,230 of them, or 76.2%.16 Nearly all of those devices, 96.2%, reached market through the 510(k) pathway by showing substantial equivalence to an existing device.16 The single most common product code is automated radiological image-processing software.16 The pace keeps rising: 335 AI devices were authorized in 2025, a record, and 181 in the first half of 2026.16

In other words, regulated AI in medicine today mostly reads images: triage, detection and image reconstruction. Aidoc, for example, received FDA clearance in January 2026 for what was described as the first foundation-model-powered clinical AI device, a CT triage tool covering 14 conditions.19 In June 2026 the FDA granted breakthrough designation, which is not market authorization, to generative AI tools that draft chest X-ray reports.12 MAI-DxO is a different kind of product: a general diagnostic reasoner that orders tests across many specialties. Nothing in the current device list looks like it.

A regulatory path that doesn't exist yet

This is where the result runs into regulation. The FDA's January 2026 final guidance on clinical decision support did not broadly exempt diagnostic AI. It stressed that clinicians must be able to independently review the basis for any recommendation.22 A tool that interviews patients directly and makes diagnoses with little human review probably falls outside that exemption.29 Commentators have floated alternatives, including a licensure-style framework for autonomous clinical AI and former FDA commissioner Scott Gottlieb's proposal to oversee these tools through laboratory certification rather than review of each device.22 There are early signs of movement. UpDoc received what was described as the first clearance for patient-facing, LLM-based software as a medical device.24 ARPA-H's ADVOCATE program expects its performers to pursue FDA authorization for clinical AI agents within 24 months of award.22

States are moving faster than federal agencies, and mostly in the opposite direction from autonomy. Texas requires practitioners to personally review AI-generated recommendations and tell patients when AI is used in diagnosis.25 A California law taking effect in 2027 bars facilities from deploying AI to independently perform functions that require a license.23 A recent systematic review warns that hospitals are adopting AI faster than this fragmented oversight can handle. It cites a chest X-ray model whose performance dropped from above 0.95 AUC on open datasets to about 0.70 once deployed in hospitals.28

The real test is the hospital study

Microsoft's next step is the one that counts. With Beth Israel Deaconess, it plans a prospective study in which patients interact with the system directly and its conclusions are compared with physicians' independent assessments.8 The company is also open-sourcing parts of SDBench under an MIT license, which should allow outside researchers to test its claims.8

Until those results come in, the fairest summary is that Microsoft has shown frontier language models, organized well, can work through difficult text cases better than individual generalists. That is a meaningful research result. Whether it holds up with real patients, imaging and incomplete information, and how a tool like this would be regulated, are still open questions.