Microsoft AI beats doctors on hard cases in Science study

Microsoft's ↘ 0.24% AI diagnostic system correctly solved 83% of complex New England Journal of Medicine cases in a simulation. Generalist physicians using reference materials got 36% right, according to a study published Thursday, October 22, in Science. Microsoft AI CEO Mustafa Suleyman called the result a step toward "medical superintelligence." He also said the tool has not yet been proven on real patients. "It needs clinical validation," he told The Verge. "That is a leap, but this is a major milestone along that journey."theverge.com+1

How the system performed

The system, called MAI-DxO, coordinates several AI agents that work like a panel of doctors. They question a patient, order tests and debate a diagnosis. Physicians working without references scored about 22% on the same cases. On emergency department cases drawn from Beth Israel Deaconess Medical Center admissions, the system scored 80%, compared with an average of 47% for physicians, usually at a lower estimated cost.techcrunch.com+1

The comparison has limits. Cases this complex are usually handled by specialists or teams of doctors, and neither group was included. The study's own data also suggest that much of the gain comes from the underlying AI models. Running GPT-5 alone with a basic prompt scored 78.9% on the complex cases, and adding MAI-DxO raised accuracy by about four points while cutting the estimated cost by about $3,000 per case. On emergency cases, the orchestrator added little. It helped weaker models most: Gemini 2.5 Flash from Google Alphabet Inc. ↗ 0.84% went from 46.4% to nearly 69%. The paper does not test the newest frontier models.theverge.com+1

From preprint to hospital trials

Microsoft first described MAI-DxO in June 2025. At the time, it said the system diagnosed up to 85.5% of benchmark cases, compared with 20% for 21 physicians who worked without outside resources. Those results came from a preprint that had not yet been peer reviewed. Microsoft now plans to open-source its SDBench benchmark.ar5iv.labs.arxiv.org+2

The next step is a study at Beth Israel. Patients will receive standard care, and those who consent will also interact with MAI-DxO so its recommendations can be compared with the doctors' decisions. "We think this will be the biggest study of its kind," Microsoft AI health vice president Dominic King told TechCrunch. He said it will likely begin "in the next couple of months." Microsoft has not said when the tool might become commercially available.theverge.com+1

Wider push into health

Suleyman said Microsoft is working with Mayo Clinic on a healthcare foundation model planned for release next year. That puts Microsoft alongside Google and OpenAI, which have also claimed their AI can match or beat doctors. "We have to have the absolute number one state-of-the-art medical clinical decision support products over the next few years," Suleyman said.techcrunch.com