Microsoft AI Diagnosis Tool Beats Generalists in Science Study

What Microsoft published

Microsoft's diagnostic AI research has now passed peer review. On October 22, Science published "Sequential Diagnosis with Language Models," the paper behind the Microsoft AI Diagnostic Orchestrator, or MAI-DxO. The work first appeared as a non-peer-reviewed arXiv preprint the previous year.2 Most medical AI demonstrations hand a model a complete case and ask for an answer. This study works differently. The system gets only a chief complaint and has to decide which history to take, which exam findings to look for and which tests to order, updating its working hypotheses as each result comes back.4

The headline numbers are large. On 304 hard clinicopathological cases from The New England Journal of Medicine, MAI-DxO reached the correct diagnosis 83.2% of the time. Generalist physicians who could use reference materials scored 36.1%.2 Without references, the physicians fell to about 22%.2 The paper also added a second set of 325 emergency department admissions from the MIMIC-IV-ED database. There the system scored 80.3% and the physicians 46.8%.5

Microsoft presents the result as a step toward what AI chief Mustafa Suleyman calls "medical superintelligence," a goal that reaches past diagnosis into treatment decisions and care coordination.7 Suleyman also predicted "extremely impressive deployments of intelligent systems in production" next year.9 The paper itself is more cautious, and that gap is the real story.

How the orchestrator works

MAI-DxO is not a new foundation model. It is a coordination layer that sits on top of existing language models.36 It splits the diagnostic job into roles. One agent maintains the differential diagnosis, one picks tests, one argues against the leading hypothesis, and one keeps an eye on cost. Microsoft says this structure counters biases such as anchoring on a first impression.210 The benchmark built around it, SDBench, uses a "gatekeeper" model that releases case details only when they are specifically requested. Scoring counts both accuracy and the estimated cost of the workup.1

The peer-reviewed version used OpenAI's GPT-5 for the main evaluation.7 The orchestration layer added an average of 7.9 percentage points of accuracy on the NEJM cases and 4.4 points on the emergency cases across the base models tested.26 One detail gets less attention than it should. GPT-5 on its own scored 78.9% on the NEJM cases, against 83.2% with the orchestrator.2122 In other words, the underlying model does most of the work. The orchestration adds a real but modest improvement on top.

Where the coverage agrees and where it splits

The accuracy figures are reported consistently everywhere. Coverage diverges on two points: how the result compares with the preprint, and cost.

The June 2025 preprint, which ran on OpenAI's o3, reported 80% accuracy against a physician average of 20%.1 In that version, physicians were barred from using search engines and other online references. The Science paper tested doctors both with and without those tools, and the gap shrank, though it remains wide.25 Critics had pointed out after the preprint that blocking standard tools like UpToDate and PubMed put the human participants at a disadvantage.26 The peer-reviewed design answers part of that objection. The 36.1% open-book figure is the fairer comparison, and the one readers should remember.

Cost is where the messaging departs furthest from the data. Microsoft's blog says the system reaches the right diagnosis "at half or even one third of the cost."2 Several detailed breakdowns tell a more complicated story. With no budget limit, MAI-DxO spent more per case than physicians did: about $5,431 versus $4,725 on the NEJM cases, and $4,979 versus $3,400 on the emergency cases.29 The savings appear only when a budget cap is applied. Under a cap, the system reached 79.6% accuracy on the NEJM cases at about $1,303 per case, while emergency-case accuracy fell to 73.2% at $2,634.2629 These are also estimated test prices inside a simulation, not savings measured in any hospital.9

The reasonable reading is that the cost claim is true only for a chosen operating point. The best-accuracy configuration was more expensive than the doctors.

What the study does not show

The authors state the limits clearly. They write that the comparisons "do not establish superiority over specialists, multidisciplinary teams, or real-world clinical practice."25 The cases were text only, with no imaging, and none involved a healthy patient.25 Because every NEJM case had a real pathological diagnosis and benign visits were excluded, the benchmark cannot measure how often the system would raise false alarms in a normal patient mix.2729 Physicians worked through a 56-case subset rather than all 304 cases.25 All of the authors are Microsoft employees who hold equity in the company.25

The study also says nothing about patient outcomes. The gatekeeper model stood in for real patients, so it could not reproduce the confusion, incomplete histories and miscommunication of a real consultation.9 The researchers themselves expect the gap to narrow if the system were compared against specialists instead of generalists.2 Some behavioral findings are interesting in their own right. MAI-DxO asked an average of 25.5 questions before ordering its first test, compared with 5.6 for physicians. When it got a case wrong, it usually named a closely related diagnosis at the wrong level of specificity.29

The next step is a prospective study with Beth Israel Deaconess Medical Center in Boston. It will start with AI-assisted triage for patients in the emergency department waiting room.23 Patients will continue to receive standard care throughout.24 That study, not the Science paper, is the first test that could show clinical value.

The regulatory question

If MAI-DxO moves from research toward deployment, its regulatory route is unclear, and its proposed first use is in a sensitive spot.

The FDA's clinical decision support guidance, finalized in January 2026, keeps the four-part test that decides whether clinician-facing software counts as a medical device. The key criterion is whether a clinician can independently review the basis for a recommendation instead of relying mainly on the software.39 Several legal analyses note that the guidance says nothing about generative AI and large language models.3234 The FDA has said separately that an LLM-based tool may meet the review criterion if clinicians can actually inspect the basis for its output.31 The agency also does not consider critical, time-sensitive decisions to meet that criterion.36 Emergency triage is close to the textbook example of a time-sensitive setting. A diagnostic orchestrator used in an emergency department would therefore likely be treated as a device rather than exempt decision support.

The FDA's thinking on generative AI is still forming. In August 2026 the agency released a discussion paper on generative AI-enabled devices, with comments due October 19. The paper asks whether agentic systems need additional oversight, but it is explicitly not guidance.3431 The agency has also said it will look at ways to tag devices that include foundation models such as LLMs on its public list.37

The radiology contrast

The FDA's list of AI-enabled devices shows where regulated clinical AI actually exists today. As of the September 4, 2026 update, the list contained 1,614 authorized devices. Of those, 1,230, or 76%, were radiology devices, and 96% reached the market through the 510(k) pathway.18 The FDA authorized 92 AI devices in the first quarter of 2026 and 89 in the second.16 Most are narrow, single-task tools: CT triage, image reconstruction, nodule scoring. Software that analyzes medical images is a regulated device by definition.33

The frontier is moving toward broader models. In January 2026, Aidoc received clearance for what was described as the first foundation-model-powered clinical device: a body CT triage tool covering 14 conditions.19 Even so, a multi-condition imaging triage tool is a much narrower claim than a general diagnostic reasoner. MAI-DxO, meanwhile, has not been tested on imaging at all.22

The takeaway

The Science paper's main contribution is a better kind of exam, more than a better diagnostic tool. Scoring AI on step-by-step information gathering and cost is more meaningful than multiple-choice medical licensing questions, and Microsoft says it will open-source the benchmark.4 The peer-reviewed results are more modest than the preprint's, and that strengthens them.

Still, an 83% score on curated teaching cases, measured against solo generalists and driven mostly by GPT-5 itself, is not evidence that patients will benefit. The "cheaper" claim depends on settings Microsoft chose. The first planned real-world use, emergency triage, falls in exactly the time-critical category where the FDA has signaled that the exemption for decision-support software is unlikely to apply. Suleyman's timeline for deployment next year looks optimistic. The Beth Israel Deaconess results will be the first real evidence of whether the system helps patients.