Microsoft’s AI diagnostic system, MAI-DxO, correctly diagnosed 83% of complex medical cases in a simulation, more than double the score of generalist doctors given the same cases, according to a peer-reviewed study published on Thursday in Science. Mustafa Suleyman, the chief executive of Microsoft AI, called the result a step toward what the company calls medical superintelligence.
MAI-DxO, short for Microsoft AI Diagnostic Orchestrator, does not work like a single chatbot answering a question. It coordinates several AI agents that behave like a panel of doctors: they question the patient, decide which tests to order, propose diagnoses and challenge each other’s reasoning over several rounds before settling on an answer.
How MAI-DxO And The Doctors Were Tested On SDBench
Microsoft built its own benchmark, SDBench, from 304 complex case reports published in The New England Journal of Medicine, plus a second set of more ordinary emergency department visits drawn from MIMIC-IV-ED, a database of admissions at Beth Israel Deaconess Medical Center in Boston. Each case begins with only a short description of the patient. The system, or the doctor, must then ask for each piece of information it wants, from the patient’s history to a scan, and every test ordered adds to an estimated bill.
On the complex journal cases, MAI-DxO scored 83.2%. Primary care physicians and general internists working alone scored 36.1% when they could use textbooks and the internet, and 22.3% without them. On the emergency department cases, MAI-DxO scored 80.3% against 46.8% for the physicians. The comparison comes with a caveat the researchers state plainly: cases this difficult would normally be handled by specialists or teams of doctors, and neither was part of the test.
What The Orchestration Adds, And What It Costs
Much of the accuracy comes from the underlying model. GPT-5 on its own, with a basic prompt and no orchestration, scored 78.9% on the complex cases at an estimated $8,409 in tests per case. MAI-DxO raised that by about four points while cutting the bill by roughly $3,000. Its most accurate setting averaged $5,431 per case, against $4,724 for doctors using references, and a budget-capped setting scored 79.6% at $1,303 per case.
The orchestration helps most where the base model is weakest. With the newer GPT-5.5, MAI-DxO lifted accuracy on the complex cases from 72% to nearly 87%, though on emergency cases both scored about 85% either way. Gemini 2.5 Flash rose from 46.4% alone to nearly 69% inside the panel. Standalone scores for other models tested included Grok 4 at 76.6%, Claude 4.1 Opus at 74.0% and Gemini 2.5 Pro at 73.0%. Microsoft tested the system with models from OpenAI, Google, Anthropic, Meta and DeepSeek, and is open-sourcing SDBench for a selection of the journal cases so other researchers can check the results.
From Benchmark To A Live Hospital Study
Microsoft says the real test comes next. Through a partnership with Beth Israel Deaconess Medical Center, the company will evaluate MAI-DxO in live hospital settings: consenting patients will receive standard care while also engaging with the system, and its recommendations will be compared with their doctors’ independent assessments across emergency and urgent care environments. Dominic King, Microsoft AI’s vice president of health, said the study will likely get underway in the next couple of months and called it potentially the biggest study of its kind.
Suleyman also told TechCrunch that Microsoft is working with the Mayo Clinic to train a foundation model for healthcare, planned for release next year, and predicted a “coding style breakthrough” for medical records in 2027 as models begin streaming predictions about how a patient’s record might unfold. The paper’s authors note their own limit in the study summary: the findings come from a controlled benchmark setting, however promising they look for lower-cost, more accurate diagnosis.
Why It Matters
MAI-DxO matters because a peer-reviewed Science paper carries weight that a company benchmark does not, and because Microsoft is framing diagnosis as the beachhead for its own frontier AI programme rather than a feature licensed from a partner. The system is model-agnostic — it improved every lab’s models it was wrapped around — which makes the orchestration layer itself the product Microsoft can sell alongside the dictation and radiology tools it already offers clinicians. For rival labs, including OpenAI, whose models powered the best results and which is widening their distribution, the study is evidence that the contest in medical AI is shifting from raw model scores to the systems built around them. The open questions are the ones the benchmark cannot answer: whether the gains survive contact with real patients, real time pressure and specialists working in teams.
Background
Microsoft first posted this work as a preprint in June 2025, when MAI-DxO ran on OpenAI’s o3 model and scored up to 85.5% against 20% for 21 doctors who were not allowed to look anything up; the published version runs on GPT-5, adds doctors with full reference access and adds the emergency cases, which is why the headline comparison is now 83% against 36%. Suleyman, a DeepMind co-founder, joined Microsoft in 2024 to lead its consumer and health AI push, and King followed from the DeepMind Health venture they built together. The effort sits alongside Microsoft’s wider agent-security and enterprise work, from its Guardian Agents for policing companies’ own AI agents to Copilot deployments reaching hundreds of thousands of employees.
Source: TechCrunch, MadRobot
0 Comments