Microsoft MAI-DxO Tops Generalist Doctors 83% to 36% in Science

A benchmark result goes through peer review

Microsoft's AI Diagnostic Orchestrator, known as MAI-DxO, has moved from a company blog post to a peer-reviewed paper. The study appeared in Science on October 22, 2026, and its results are both stronger and more carefully hedged than the claims that made the system famous in mid-2025.57 The main numbers come from 304 difficult case records published in the New England Journal of Medicine. Running on OpenAI's GPT-5, MAI-DxO reached the correct diagnosis in 83.2% of those cases. Generalist physicians managed 36.1% when they could use search and reference tools, and 22.3% when they could not.721

The paper adds a test the 2025 preprint did not have: 325 emergency department admissions taken from the MIMIC-IV-ED dataset. On those cases the orchestrator scored 80.3% and the physicians scored 46.8%.522 Microsoft also released its Sequential Diagnosis Benchmark (SDBench) on GitHub under an MIT license and announced a clinical study with Beth Israel Deaconess Medical Center. The orchestrator itself is still unavailable as a product, API or download, and it is not approved for clinical use.721

The main lesson of this story is that a benchmark has been validated, not a medical device. The difference matters a great deal for regulation, hospital purchasing and radiology.

How the system works, and what changed since 2025

MAI-DxO is an orchestration layer that runs on top of existing language models, not a new model. One model plays five panel roles: Dr. Hypothesis, Dr. Test-Chooser, Dr. Challenger, Dr. Stewardship and Dr. Checklist. Together they track a list of possible diagnoses, choose the next test, push back on early conclusions and keep an eye on cost.7 It starts with a short case summary and has to ask for every further detail. A separate "gatekeeper" model holds the full record and answers only what a real clinician could reasonably find out.910

The headline numbers have shifted since the original announcement. In June 2025 Microsoft reported 85.5% accuracy against an average of 20% for 21 physicians who worked with no outside resources at all.68 The peer-reviewed version gives physicians reference tools and puts their score at 36.1%. That narrows the gap from roughly fourfold to about 2.3 times on the NEJM cases, and about 1.7 times on the emergency cases.22 The old figures are still in circulation. As late as October 24, 2026, at least one trade outlet was still leading with 85.5% versus 20%.3 Anyone quoting "four times better than doctors" is quoting a result the authors have since replaced.

The working styles also differed. On average the AI asked 25.5 questions before ordering its first test, while physicians asked 5.6. When the AI got a case wrong, it usually named a closely related diagnosis at the wrong level of detail. Physician errors tended to be further from the answer.24

Where the reporting diverges: cost and attribution

Coverage disagrees most on cost. Microsoft has argued since 2025 that the orchestrator is both more accurate and cheaper than physicians, and Reuters' account of the new paper repeated that message.624 A closer reading of the paper is less flattering. With no budget limit, average testing costs per case were higher for the AI: $5,431 against $4,725 on the NEJM cases, and $4,979 against $3,400 on the emergency cases. Savings only appeared once budget caps were set. With caps, accuracy dropped to between 79.6% and 80.9% on the NEJM cases and to as low as 69.5% on the emergency cases.2422 A press release from the American Association for the Advancement of Science gave average simulated costs ranging from $1,303 to $8,945 per case, depending on the setting.21 The fairest summary is that MAI-DxO lets a health system pick its own trade-off between cost and accuracy, not that it saves money in every case.

The second dispute is about how much of the gain comes from Microsoft's design. GPT-5 on its own scored 78.9% on the NEJM cases, only a few points below the orchestrated 83.2%.22 One analysis concluded from the paper's own data that stronger base models produce much of the improvement, and that orchestration helped weaker models most.10 An independent review on Nabu Science rated the paper "Limited, with caveats" and gave it 2.0 out of 5 for methodological rigor. The review noted that the orchestration gains, 7.9 points on the NEJM cases and 4.4 on the emergency cases, were measured against minimally prompted versions of the same models.21 The same review pointed to possible contamination: the NEJM cases are public text that GPT-5 may have seen in training.21 The 2025 preprint already admitted that memorization could not be ruled out.26

The study also had built-in limits. The cases were text only, with no physical exam, imaging, waveforms or pathology slides. Every case had a confirmed pathological diagnosis, so there were no healthy or benign presentations. The human comparison was individual generalists, not specialists or teams.2422 Because of that case mix, the study could not measure false-positive rates. That is a serious gap for any tool meant for routine care.26 Dominic King, a Microsoft AI vice president and one of the authors, has acknowledged the artificial setup himself: no patient ever arrived with a tidy summary.24

The regulatory question: device or decision support

Microsoft says the next steps are real-world trials and finding a regulatory pathway.3 Mustafa Suleyman, who leads Microsoft AI, has suggested that production-grade diagnostic systems could appear as early as next year.5 That timeline runs into a US framework that has not yet decided how to treat generative AI.

The FDA's clinical decision support guidance, issued in January 2026 and reissued later that month, now allows software to give a single recommendation when only one option is clinically appropriate. The software can still stay outside device regulation, as long as a clinician can independently check the basis for that recommendation.3531 The guidance text does not mention generative AI at all.3233 At a March 2026 town hall, FDA staff said an LLM-based function could meet the independent-review test if it gives clinicians enough to review. The guidance also says that software supporting critical, time-sensitive decisions generally fails that test.33

In this analysis, that last point is the one that matters most for MAI-DxO. Its strongest new evidence comes from emergency department admissions, which is exactly the time-pressured setting where the decision-support exemption is weakest. A system that names a specific diagnosis for any input is also hard to fit into the non-device category. Legal analysts have warned that opaque, directive or time-critical uses should still expect device oversight.3435 MAI-DxO's step-by-step record of questions and tests could help with the transparency requirement. Whether a long transcript of debate among model personas counts as a basis a clinician can review is a question regulators have not answered.

The FDA's August 18, 2026 discussion paper on generative AI devices is the closest thing to a roadmap. It sketches a two-axis risk framework and a premarket "competency assessment" loosely modeled on how physicians are trained and evaluated, combining benchmark testing with clinical confirmation. It also asks specific questions about foundation models and agentic AI. Comments closed on October 19.3633 SDBench looks a lot like the non-clinical benchmarking piece of that idea. The Beth Israel study would need to supply the clinical confirmation.

Why radiology shows the realistic path

The current market suggests that diagnostic AI will reach patients one narrow step at a time, not all at once. The FDA's list counted 1,614 AI-enabled devices through June 2026. Radiology accounted for 1,230 of them, or 76%.2017 Of the total, 96.2% were cleared through the 510(k) pathway, with only 40 De Novo grants and 21 premarket approvals.17 These devices have narrow, imaging-specific uses, such as triage software for one class of findings. Even the January 2026 clearance described as the first foundation-model-powered clinical device covers a defined set of 14 CT conditions.19

MAI-DxO is the opposite kind of system: open-ended, multi-specialty, built on general-purpose models that change often, and so far tested without any images. Only two devices on the FDA's list mention a predetermined change control plan, the mechanism that lets a cleared AI device be updated without a new submission.17 That is a weak base for a product whose own study showed results tracking the underlying model.

What the evidence supports

The Science paper is a real advance in how diagnostic AI is evaluated. Testing diagnosis as a sequence of questions and tests, with costs attached, is more meaningful than multiple-choice exams, and releasing SDBench lets others check the work.710 Taken together, though, the coverage supports a narrower claim than the headlines. A GPT-5-based system outperforms individual generalists on difficult, text-only teaching cases. Its cost advantage depends on how it is configured. And the orchestration layer adds modest gains on top of a frontier model that already does most of the work.

Hospitals considering clinical AI should treat the Beth Israel study as the result that actually matters. Regulators should watch whether the FDA's competency-assessment idea becomes formal guidance before Suleyman's timeline arrives. Until a prospective trial reports patient outcomes, MAI-DxO remains what Microsoft itself calls it: a research system that has not been cleared for the bedside.2125