AI Medical Diagnosis: OpenAI Model Beats Doctors on Real ER Cases

What the study found

For years, claims that AI beats doctors at diagnosis rested on tidy material: multiple-choice exams, textbook vignettes, and hand-picked case reports. A study published in Science on April 30, 2026 changed that. A team led by physicians and computer scientists at Harvard Medical School and Beth Israel Deaconess Medical Center (BIDMC) ran OpenAI's models against working physicians on real emergency department records.3234

The central experiment used 76 patients who came through the Beth Israel emergency room. The researchers compared diagnoses from two internal medicine attending physicians with diagnoses from OpenAI's o1 and 4o models. Two other attendings graded the results without knowing which answers came from a human and which came from a machine.32 Each case was assessed at three points: initial triage, the first evaluation by a doctor, and admission to the floor or the ICU.34 The team says it did no cleanup on the records. The model saw whatever was in the electronic chart at that moment, just as the physicians did.35

At triage, where information is thinnest, o1 reached the exact or a very close diagnosis in 67% of cases. The two physicians managed 55% and 50%.32 The gap narrowed as more information came in, but the model still led by about 2 to 10 percentage points later in the visit.31 The study also covered published case material. On 143 complex New England Journal of Medicine cases, o1-preview put the correct diagnosis in its differential 78.3% of the time. A previously published physician baseline, where doctors could use search engines and references, scored 44.5%.33

Co-author Adam Rodman said the main takeaway for him was that the model holds up on the messy real-world data of an emergency department.34 Co-author Arjun Manrai said the model beat both earlier models and the physician baselines on nearly every benchmark the team tried.32

Where the coverage diverges

Outlets agree on the numbers. They split on what the numbers mean, and the split matters.

The most pointed criticism is about who the AI was compared against. Emergency physician Kristen Panthagani called it an interesting study that had produced overhyped headlines. Her point is that the doctors were internal medicine attendings, not emergency physicians, and a fair test should use specialists from the relevant field.35 She also argued that an ER doctor's first job is not to guess the final diagnosis but to rule out anything that could kill the patient.32 TechCrunch updated its own story to say the physicians were internists.32 Some coverage still billed the result as a win over "ER doctors" or "expert" emergency physicians, which goes beyond what was tested.38

Other limits come from the authors themselves or from close readers. The study was retrospective. The AI never guided real care.36 On rare but critical "cannot-miss" diagnoses, the model was not significantly better than the physicians, and that is the category where an AI error would do the most damage.36 Co-authors also told reporters that the model sometimes suggested unnecessary tests.39 The model worked from text only, with no imaging, even though scans are central to diagnosing things like blood clots and cancers.31 Rodman has also warned that performance would probably drop for hospitalized patients with days of accumulated records, compared with a short ER visit.31

One factor works the other way. o1 came out in late 2024, which co-author Thomas Buckley called "ancient history" in machine-learning terms.31 Stanford internist Eric Strong, who was not involved, said the model's age doesn't matter much because newer systems are likely to do as well or better.31

Our reading is that the study does not show AI is a better doctor. It shows something narrower that still matters for deployment: a general-purpose reasoning model is no longer clearly worse than a capable physician at reading a raw chart, and it may help most at the most rushed, information-poor moment of the visit. The authors' own conclusion is that prospective trials are urgently needed, not that the model is ready to use.32

Microsoft's follow-on and the benchmark race

In late October, a second Science paper made a stronger claim. Microsoft's AI Diagnostic Orchestrator (MAI-DxO) simulates a panel of clinical experts and gathers information step by step instead of reading the whole case at once.2 It scored 83.2% on 304 NEJM clinicopathological cases and 80.3% on 325 emergency department cases from the MIMIC-IV-ED database. Generalist physicians scored 36.1% and 46.8%.5 The system also asked many more history questions before ordering its first test, averaging 25.5 against the doctors' 5.6.8 Its tests were estimated to cost less. In one setup it reached about 81% accuracy at roughly $2,241 per case, while the physicians averaged about $4,725.6

The caveats are similar, and arguably larger. The cases were simulated. An automated gatekeeper fed information to both the AI and the doctors.10 The researchers themselves wrote that the comparisons do not show superiority over specialists, multidisciplinary teams, or real clinical practice.3 One analysis noted that much of the work is done by the underlying model. GPT-5 alone solved about 79% of the complex cases, and the orchestration layer raised that to around 83%.6 For buyers and regulators, that means a product's performance may depend heavily on which commercial model sits underneath it, and those models change often.

Both research efforts now lead to the same hospital. Microsoft plans a clinical test with BIDMC to evaluate triage in emergency department waiting rooms.2 Patients there will receive standard care while researchers compare the AI's recommendations with doctors' judgments, and doctors will keep making treatment decisions.6 This is the kind of prospective work the Harvard authors asked for.

The regulatory gap

The biggest open question may be regulatory. According to a June 2026 Congressional Research Service brief, the FDA appeared not to have authorized any generative AI-enabled device. It had granted breakthrough designation, which is not market authorization, to a patient-facing generative AI app from RecovryAI.22 The agency later gave breakthrough designation to two generative tools that draft chest X-ray reports.25

The FDA's AI-Enabled Medical Device List reached 1,614 entries in its September 4, 2026 update.25 About 96% of those devices were cleared through the 510(k) pathway, which relies on similarity to existing products.29 Most are radiology tools that detect or measure specific findings.25 An open-ended diagnostic reasoner like o1 or MAI-DxO does not fit that mold well.

At the same time, the agency is easing oversight in some areas. In January 2026 it relaxed its approach to clinical decision support software. Some tools that give a single recommendation can now reach the market without FDA review, as long as they meet the agency's other criteria.27 Law firm McDermott described the change as a more hands-off approach that lets more technology reach the market without premarket review.28 STAT reported that the change could let unregulated generative AI tools into clinical workflows.27

Other efforts point toward more oversight. The FDA's draft lifecycle guidance would expect manufacturers to disclose training-data demographics and monitor real-world performance for model drift.21 Congress directed the agency to report on whether it needs new authority to oversee AI devices after deployment.22 The FDA also took public comment on how to regulate generative and agentic AI devices, in a docket that closed October 19, 2026.29

Liability is unresolved too. Rodman has said there is no formal accountability framework for AI diagnoses.32 He compared the likely legal position to today's clinical decision support tools, where the doctor stays responsible for the decision.33

What hospitals are actually deploying

Hospitals already use AI widely, but mostly for other tasks. One industry compilation estimates that about 80% of hospitals use AI somewhere, and fewer than 20% report sustained, highly successful use in core clinical diagnosis.11 The tools that have spread are documentation tools like Microsoft's DAX Copilot, which runs in more than 150 health systems, while diagnostic language models remain largely in research.14 Clinical AI that hospitals do deploy tends to be narrow and FDA-cleared, such as Advocate Health's imaging models, which are used with oversight from trained clinicians.15

The two Science papers make it harder to argue that language models can't reason about diagnosis. They don't show that these models improve patient outcomes, and no one has shown that yet. The BIDMC triage trial will be the real test. Meanwhile, the evidence is moving faster than the rules that would govern these tools in hospitals.