Microsoft AI claims 4x diagnostic accuracy over doctors in Science study
Microsoft's Diagnostic Orchestrator, an AI system designed to mimic medical expert panels, demonstrated diagnostic accuracy up to four times higher than human physicians in a study published in Science. The tool, developed under CEO Mustafa Suleyman, achieved 83% accuracy on complex cases compared to 36% for generalist doctors. Microsoft plans to open-source the SDBench benchmark, marking a significant step in AI's application to clinical diagnosis and potentially reshaping healthcare standards globally.
Key points
- Microsoft AI CEO Mustafa Suleyman announced that the company's Diagnostic Orchestrator outperforms human doctors in diagnosing medical conditions, with findings published in the journal Science.
- The AI system achieved approximately 83% diagnostic accuracy on cases derived from The New England Journal of Medicine, compared to roughly 36% for generalist physicians using reference materials.
- In emergency department scenarios from Beth Israel Deaconess Medical Center, the AI scored 80% accuracy against an average of 47% for human doctors.
- Microsoft developed SDBench, a benchmark compiled from medical case reports and emergency data, which the company intends to open-source to the wider research community.
- The Diagnostic Orchestrator utilizes multiple AI agents to reason through symptoms, order tests, and arrive at diagnoses, mimicking the investigative process of a panel of medical experts.
- The study positions AI as a viable tool for improving diagnostic precision, a sector long touted by tech leaders as a primary area for technological life improvement.
Microsoft has released findings indicating that its new AI-powered diagnostic tool, the Diagnostic Orchestrator, significantly outperforms human physicians in identifying medical conditions. The results, published in the prestigious journal Science, show the system achieving diagnostic accuracy rates up to four times higher than generalist doctors. Mustafa Suleyman, Microsoft’s AI CEO, described the development as a long-awaited milestone for artificial intelligence in healthcare.
The study evaluated the AI against human doctors using a series of scenarios drawn from medical case reports and emergency department data. On complex cases sourced from The New England Journal of Medicine, the Diagnostic Orchestrator recorded an accuracy rate of approximately 83 percent. In contrast, generalist physicians using reference materials achieved roughly 36 percent accuracy, while those working without references scored 22 percent. The AI also demonstrated superior performance on emergency department cases from Beth Israel Deaconess Medical Center in Boston, scoring 80 percent compared to an average of 47 percent for human doctors.
The system is designed to function like a panel of medical experts, employing multiple AI agents to reason through symptoms, determine necessary tests, and formulate diagnoses. To support future research, Microsoft compiled the test cases into a benchmark called SDBench, which it plans to open-source. This move aims to standardize the evaluation of AI diagnostic capabilities, potentially accelerating the integration of advanced machine learning into clinical workflows worldwide.
Sources
The WireByte editorial team synthesises technology news from multiple primary sources, verifies the facts, and links every source. Articles are produced with AI assistance and reviewed under our editorial policy.
