AI Diagnostic Panel Outperforms Generalist Doctors 83% to 36% in Complex Case Study

The rapid advancement of artificial intelligence in healthcare has reached a critical milestone, with a new system demonstrating capabilities that challenge traditional clinical workflows. A study published this week in the journal Science reveals that Microsoft's AI diagnostic tool, MAI-DxO, significantly outperformed human generalists in identifying complex medical conditions.

The core of the finding is a stark statistical disparity. In a simulation using difficult cases from the New England Journal of Medicine, the AI system achieved an accuracy rate of 83%. In contrast, primary care doctors and general internists, working individually with access to reference materials, were correct only 36% of the time. It is important to note that these specific cases are typically handled by specialist teams in real-world settings, who were not included in the direct comparison.

MAI-DxO operates by coordinating multiple AI agents that function similarly to a panel of physicians. These agents interrogate the patient, order necessary tests, and engage in internal debate to reach a consensus diagnosis. This multi-agent approach is designed to mimic the collaborative nature of medical decision-making. The system was tested using large language models from various providers, including OpenAI, Google, Anthropic, Meta, and DeepSeek. The most robust results were obtained when the system utilized OpenAI's GPT-5 and GPT-5.5 models.

The data suggests that the majority of the performance improvement stems from the underlying power of the AI models themselves, with the orchestration layer providing additional benefits. When GPT-5 was used with a basic prompt, it scored 78.9% on complex cases at a cost of approximately $8,400 per case. The addition of the DxO orchestration reduced this cost by about $3,000 while improving accuracy by roughly four points. However, on emergency department cases, the system showed minimal improvement in both accuracy and cost efficiency compared to the base model.

With the newer GPT-5.5 model, released in April of this year, the benefits of the orchestration became more pronounced for complex cases. Accuracy rose from 72% to nearly 87% at a slightly lower cost. Yet, this pattern of limited gain on emergency cases persisted. The system proved most effective when paired with less capable models, such as Gemini 2.5 Flash, where accuracy jumped from 46.4% to nearly 69%. Despite this boost, the gap between weaker and stronger models remained larger than the improvement provided by the system itself.

Mustafa Suleyman, CEO of Microsoft AI, characterized the results as a significant step toward medical superintelligence. He noted that the company is currently partnering with the Mayo Clinic to train a dedicated foundation model for healthcare, with a release expected next year. Suleyman expressed confidence that by 2027, the company will approach a breakthrough in processing medical records, allowing for highly accurate predictions of how a patient's condition might evolve. He described this foresight as invaluable for clinicians.

Suleyman, a co-founder of DeepMind who joined Microsoft in 2024, has long advocated for AI in medicine. He predicts that healthcare models will become an order of magnitude more capable within the next 24 to 36 months. He explained that this progress relies on a flywheel effect, where models in production are iteratively improved based on real-world usage to identify and patch gaps. This strategy has previously driven rapid advancements in coding and chat applications.

To validate these findings in practical settings, Microsoft has established a partnership with Beth Israel Deaconess Medical Center in Boston. The collaboration will evaluate the clinical reasoning of MAI-DxO across various hospital environments. Patients will continue to receive standard care, while those who consent will also interact with the AI system. Microsoft and its partners will then compare the AI's recommendations against the decisions made by human doctors. Dominic King, vice president of Health at Microsoft AI, stated that the study aims to cover both emergency and urgent care settings to provide compelling evidence for broader deployment.

The publication of this study coincides with a broader manifesto from Microsoft outlining its path to medical superintelligence. The company aims to move beyond simple task benchmarking and integrate AI deeply into clinical workflows. Microsoft already offers dictation and radiology tools to clinicians and views this new system as a natural expansion of its portfolio. Suleyman emphasized that providing the state-of-the-art medical decision support products is a key differentiator for the company in the competitive landscape of frontier AI labs.