Microsoft's Diagnostic AI Featured in *Science*: Beyond Benchmark Scores, It's Important to Understand How It Was Tested

October 22, 2026,Microsoft corporationAnnouncement, Diagnostic System MAI-DxO Compared to Sequential Diagnostic Benchmarks SDBench The study was officially published in *Science". Early results from this project were already released in 2025; what is noteworthy this time are the expanded evaluations, the publicly released evaluation tools, and the plans for the next phase of validation in real-world clinical settings."Microsoft's Latest Statement｜An Introduction to Early Research

This development brings one issue into sharper focus: when evaluating medical AI, it is not enough to simply determine whether it got the final answer correct; we must also examine how it arrived at that conclusion, the cost involved in reaching that answer, and whether the advantages demonstrated in testing can be translated to the clinical setting.

Change "selecting an answer" to "gradually finding evidence"

In traditional medical multiple-choice questions, all information is typically provided at once. SDBench, however, provides only a brief case summary, allowing the system under test to decide on its own what to ask next, which tests to order, and when to submit a diagnosis. MAI-DxO is one such diagnostic orchestration system; it uses multiple virtual roles to propose hypotheses, select tests, and challenge one another. According to Microsoft’s description, it updates its judgments as evidence becomes available, while simultaneously considering the value and cost of each test.Introduction to Microsoft Systems

This design offers direct insights for AI products: a system that provides the correct answer may have taken a lengthy and costly path, while a system that expresses itself fluently may have stopped asking follow-up questions too soon. Only by incorporating intermediate steps into the evaluation can we uncover the differences masked by a single accuracy metric.

However, the consultation here takes place in a simulated text-based environment. The Gatekeeper, which is responsible for providing information, reads the medical records; any missing information may be generated by the Gatekeeper. The final diagnosis is scored by the Judge against the reference answer, with scores of 4 or 5 on a 5-point scale considered correct.SDBench Official Method Description

Therefore, performance is influenced by the diagnostic system, the method of feedback, and the scoring mechanism. When comparing models, these conditions should be held constant; when evaluating real-world applications, it is important to continue examining what the simulation process may have overlooked.

Emergency cases are more relevant to everyday life, but there are still limits to their selection.

In addition to using challenging cases from the *New England Journal of Medicine*, this study also incorporated cases from the MIMIC database. This brings the scope of the evaluation closer to common clinical scenarios; however, the phrase “from the emergency department” should not be interpreted to mean “covering all emergency department patients.”

According to the official data description, this set of cases was selected from patients admitted to the hospital after visiting the emergency department; cases admitted directly to the ICU were excluded, and records with clear chief complaints, clinical information, and final diagnoses were selected. The published evaluation used a validation set of 254 cases and a test set of 71 cases, the latter of which was reviewed by clinical staff.PhysioNet Data Description

This scope affects the interpretation of the results. Cases with a clear discharge diagnosis can be used to test whether the system can follow the information trail to find the answer; real-world patient consultations also include cases with vague symptoms, incomplete records, or those for which a diagnosis cannot yet be determined. Improving performance on the former task is commendable; however, the system’s ability to handle the latter still requires supporting evidence.

Similarly, when comparing doctors, it is also important to consider what resources they have access to. In this report, Microsoft presented separate results for doctors who were and were not allowed to use external reference materials. Omitting the reference conditions and stating only that “it is X times greater than that of a doctor” would lead readers to overestimate the scope of the conclusion.Microsoft Evaluation Guidelines

"More accurate" and "more economical" should be compared under the same configuration.

Cost is a key aspect of this study, and it is also the aspect most likely to be reduced to a single slogan in communications.

The PhysioNet page provides a set of specific comparisons: among the 325 validation and test cases mentioned above, the accuracy of DxO (without budget constraints) was 80.31 TP4T, while the GPT-5 baseline was 76.61 TP4T; The average estimated cost of diagnostic tests for the two models was approximately $4,980 and $4,240, respectively. In this configuration, higher accuracy comes at the cost of increased expenses.Results and Costs for the Same Group

This suggests that when evaluating a screening system, one should consider a range of trade-offs between cost and accuracy, rather than piecing together the best results from different configurations to create a product promise that is “both the most accurate and the cheapest.” Purchasers must also ask: Is the additional testing worth it just to identify a few more cases? Should high-risk missed diagnoses and general errors be weighted the same? These are application-specific decisions that no single ranking list can answer.

The cost metrics also need to be aligned. The official code guidelines distinguish between model-based estimates and estimates based on hospitals’ published price lists; the two should not be used interchangeably. Furthermore, these figures should not be directly converted into patients’ final bills or the amounts that Chinese hospitals could save.SDBench Cost Method

Open-Source Testing Tools: Reproducibility Conditions Are Key

The open-source code allows researchers to examine the implementation of question generation, scoring, and cost calculation; however, fully reproducing the results still presents data-related barriers. The NEJM cases require separate authorization; access to the MIMIC dataset requires appropriate qualifications, training, and a data usage agreement. The five synthetic cases included in the repository are intended solely for development and testing and cannot be used to report diagnostic performance.Explanation of Data in the Official Repository

This also serves as a practical reminder for technical teams: successfully running a sample case only proves that the process is operational. To truly compare your own system, you should specify the cases used, data segmentation, model version, scoring configuration, and cost algorithm, and report on failed and incomplete cases—rather than simply showcasing successful cases.

The next step is to find out what happened in the examination room.

Microsoft announced a partnership with Beth Israel Deaconess Medical Center in Boston to conduct a prospective study in which patients will interact directly with the system, with results compared to independent evaluations by physicians. This is a future validation effort and should not be described as a completed clinical trial.Microsoft Research Program

For the general reader, here are three questions to use when evaluating the next medical AI news story: What kind of people were tested? What information and tools did the control group receive? Did the study improve scores, or did it improve observed clinical outcomes?

For the product team, the questions should be even more specific: When must the system provide results to doctors? Do incorrect recommendations increase the burden of review? Would reducing the number of tests delay critical decisions? These questions can only be answered by examining real-world workflows and patient follow-up data.

The value of MAI-DxO lies in bringing the selection of information and resource consumption during the diagnostic process to the forefront of evaluation. The key question moving forward is whether this advantage can be sustained under real-world conditions—with real people, real time constraints, and medical liability.