4 ms·
> Microsoft AI Diagnostic Orchestrator (MAI-DxO) correctly diagnoses up to 85% of NEJM case proceedings, a rate more than four times higher than a group of expe
by gm678 1y ago
> Microsoft AI Diagnostic Orchestrator (MAI-DxO) correctly diagnoses up to 85% of NEJM case proceedings, a rate more than four times higher than a group of experienced physicians.
> Clinicians in our study worked without access to colleagues, textbooks, or even generative AI, which may feature in their normal clinical practice.
1. As I understand, it's very common for doctors to fall back on reference material in their practice, especially for the most complex cases. If all access to resources was cut off (as seems to be implied by the second quote), the comparison seems somewhat unfair.
2. What were the publication dates of the case records? I can't find this information, and it makes a difference if the NEJM case studies were in the LLMs' training data.
- miraculixx 1y agoExactly. The study has been set up to produce this exact result. They essentially limited the human doctors to bare essentials, on specialist cases(!), while providing the LLMs with all sorts of help, including discussion among several AIs. That's like letting one group of students have a strict closed-book exam, while another group can take the test as a group exercise and accessing any material they like, then claiming that closed-book exams lead to worse outcomes. In a nutshell the study is just slop designed to get attention. The headline result is what they really want people to hear, and that's all the media will be repeating.