Agent Upstart

Cool things people are building with agents — and practical help getting the most out of them.

Playbooks · Coding tasks

The 83% AI diagnosis headline, read properly

Written November 4, 2026

Microsoft's MAI-DxO beat solo generalists on simulated cases in a peer-reviewed study. The benchmark is real; the clinic is not what was tested.

Scope note. This page is about reading a benchmark. It contains no medical advice and nothing that diagnoses, triages, or recommends care for any person. The study it describes tested a system on recorded cases in a simulation. It did not test care for real patients.

What the study did

In October 2026, Microsoft researchers published "Sequential diagnosis with language models" in Science. The paper introduces a benchmark, SDBench, that turns diagnosis into a stepwise encounter instead of a one-shot question. Each case starts with a brief clinical vignette. The diagnostician — an AI system or a physician — asks questions, orders tests, or commits to a final diagnosis, and the benchmark scores both accuracy and the estimated cost of the workup. A Gatekeeper agent holds the case record and reveals findings only when explicitly asked; where a requested finding is absent from the record, it generates a diagnostically neutral synthetic result so the absence itself gives nothing away. A Judge agent scores the final diagnosis against the reference diagnosis on a five-level physician-authored rubric, counting scores of 4 or 5 — diagnoses for which management would be unchanged — as correct. Costs are estimates from procedure-based pricing taken from one major health system's chargemaster.

The system under test, DxO, is an orchestrator that runs on top of a language model: it simulates a panel of clinical experts with complementary roles, revises a differential diagnosis as evidence arrives, weighs the cost of each next test, and aggregates diagnoses across independent runs. The main results pair DxO with GPT-5. The case sets are 304 clinicopathological conference cases from the New England Journal of Medicine and 325 emergency-department-to-admission encounters from the MIMIC-IV-ED dataset.

What the numbers are

Without budget constraints, DxO with GPT-5 reached 83.2% accuracy on the NEJM cases at an average estimated cost of $5,431 per case, and 80.3% on the emergency cases at $4,979. Under budget constraints the NEJM figures fell to 80.9% at $2,241 and 79.6% at $1,303, while a configuration aimed at diagnostic excellence reached 84.5% at $8,945. On the emergency cases, constrained configurations reached 73.2% at $2,634 and 69.5% at $2,037. Quote the configuration with the figure: the headline accuracy belongs to the unconstrained configuration at the highest stated cost, and the cheaper configurations trade accuracy for lower estimated cost.

The physician comparison ran as a separate study inside the paper. For the NEJM cases, 56 cases were held out from development work, and participating physicians worked an average of 36.4 cases closed-book and 34.8 open-book, giving 764 closed-book and 662 open-book diagnoses. Their average accuracy was 22.3% closed-book at an average estimated cost of $3,049, and 36.1% open-book — with search and reference tools allowed — at $4,725. The best individual physician reached 51.8%. On the emergency cases, physicians averaged 46.8% at $3,400, with the best individual at 62.5% over 40 cases. On the same cases, DxO asked a mean of 25.5 questions before ordering its first test, against 5.6 for the physicians: extensive structured history-taking before testing, rather than interleaved questions and tests.

Orchestration added to the underlying model rather than replacing it. Against a minimal prompt on the same model, DxO improved accuracy by an average of 7.9 percentage points on the NEJM cases and 4.4 points on the emergency cases. Before publication the team repeated the full evaluation with GPT-5.5 without changing prompts or orchestration; DxO still defined the cost-accuracy frontier, and its advantage over the baseline prompt grew. The paper's authors are all Microsoft employees holding equity in the company, and eight are inventors on a Microsoft patent application for an automated evaluation framework for sequential diagnosis. Peer review is not independent replication: no outside team has reproduced these results.

What the study does not show

The paper states its own boundary plainly: the comparisons do not establish superiority over specialists, multidisciplinary teams, or real-world clinical practice, and the authors expect the gap to narrow against expert specialists or teams matched to each case. The physicians in the study worked alone. The NEJM cases were chosen for teaching value and often include specialist reflections; the emergency cases are routine by comparison, and physicians did better on them, as the authors expected.

The benchmark is also a simulation with idealized conditions, in the paper's own words. Both case sets contain only pathological cases: every case had a confirmed diagnosis, and healthy or benign presentations were excluded, so the case mix does not mirror routine practice. Physicians and AI agents alike interacted with a text-based gatekeeper rather than with electronic health records or the full richness of a patient encounter. The sessions had none of the noise of real practice — incomplete documentation, patients omitting or forgetting information, changes over time — and no imaging, waveforms, or pathology slides, which the paper calls central to real diagnostic workflows. The Judge scores disease-level correctness only; it does not assess whether treatment was appropriate, whether the workup was safe, or how a patient fared over time. The cost figures are standardized pricing approximations, not money any hospital spent.

Do not carry the earlier preprint's figures into this result. An earlier version of the work, published as a preprint in June 2025, paired the orchestrator with a different model, OpenAI's o3, and reported different numbers. Those figures belong to that version. The peer-reviewed numbers are the GPT-5 figures above, with the GPT-5.5 rerun noted. Quoting the preprint's 85.5% or its physician average as the Science result misstates what was published.

What to do with the headline

If you cite this study, cite the benchmark and its conditions, not the ratio. Say which case set, which configuration, and which estimated cost the figure belongs to, and name the comparison group as individual generalist physicians working alone on recorded, text-only cases. Do not write that AI beats doctors, that diagnosis is solved, or that a diagnostic agent is ready for patients: the paper demonstrates a cost-accuracy frontier inside a simulation, and it says so itself. A builder evaluating a diagnostic tool should ask for the same things this paper supplies — the case mix, the comparison group, the cost basis, and the limits — and treat any vendor claim that omits them as unproven.

The useful lesson for builders is narrower and real. Structured sequential reasoning — tracking hypotheses explicitly, challenging the leading one, and pricing each next test before ordering it — improved on a minimal prompt on the same model, and the gain grew with a stronger model. That is evidence for building diagnosis-shaped workflows with explicit hypothesis tracking and cost-aware test selection, evaluated on stepwise cases rather than static vignettes. It is not evidence about outcomes for patients, which this study did not measure. The paper's code for the benchmark is published under the MIT License; the NEJM-derived dataset is available from NEJM Group for noncommercial research under license, and the emergency-case subset is released through PhysioNet under its credentialing and data-use terms.

Limits

Related playbooks

Sources

Back to: Coding playbooks hub · Also: The intelligence-explosion argument, and its evidence · Thousands of agents read a mouse-brain atlas for leads