4 ms·

>>This work showed that GPT-4 performance is comparable with that of physicians on official medical board residency examinations. Model performance was near or

>>This work showed that GPT-4 performance is comparable with that of physicians on official medical board residency examinations. Model performance was near or above the official passing rate in all medical specialties tested. Given the maturity of this rapidly improving technology, the adoption of LLMs in clinical medical practice is imminent. Specifics sub-scores: >GPT-4 ranked higher than the majority of physicians in psychiatry, with a median percentile of 74.7% (95% confidence interval), and it performed similarly to the median physician in general surgery and internal medicine, displaying median percentiles of 44.4% (95% CI) and 56.6% (95% CI). >GPT-4 performance was lower in pediatrics and OB/GYN but remained higher than a considerable fraction of practicing physicians, with a median score of 17.4% (95% CI) and a median score of 23.44% (95% CI). >GPT-3.5 did not pass the examination in any discipline and was inferior to the majority of physicians in the five disciplines. Overall, GPT-4 passed the board residency examination in four of five specialties, revealing a median score higher than the official passing score of 65%. ...so glad I dropped out of medskool.

azalemeth 3y ago
>GPT-4 performance was lower in pediatrics and OB/GYN but remained higher than a considerable fraction of practicing physicians, with a median score of 17.4% (95% CI) and a median score of 23.44% (95% CI). How can GPT4 perform a bimanual exam? Comfort a woman after experiencing pregnancy loss -- or help them deal with an unwanted pregnancy? Provide carefully evidenced, yet appropriate, words of reassurance or care? Deal with all of the social ignorance and generic shit that goes with this area of medicine? Medical exams essentially test rote learning with a little bit of creative thinking and problem solving involved. They are a minimum bar for competence in a doctor. They do not necessarily make for a good one.
- ProllyInfamous 3y ago
  My only response to this is that the USMLE (3-step medical licensing qualifier) has a correlary relationship between score and future ability to practice sound medicine. While in general I do agree that some physicians can exhibit exceptional bedside manner... most of the med students I started school with are so burnt-out now that they don't give a fuck about almost any of their patients (other than "don't get sued").

azalemeth 3y ago

>GPT-4 performance was lower in pediatrics and OB/GYN but remained higher than a considerable fraction of practicing physicians, with a median score of 17.4% (95% CI) and a median score of 23.44% (95% CI). How can GPT4 perform a bimanual exam? Comfort a woman after experiencing pregnancy loss -- or help them deal with an unwanted pregnancy? Provide carefully evidenced, yet appropriate, words of reassurance or care? Deal with all of the social ignorance and generic shit that goes with this area of medicine? Medical exams essentially test rote learning with a little bit of creative thinking and problem solving involved. They are a minimum bar for competence in a doctor. They do not necessarily make for a good one.

ProllyInfamous 3y ago
My only response to this is that the USMLE (3-step medical licensing qualifier) has a correlary relationship between score and future ability to practice sound medicine. While in general I do agree that some physicians can exhibit exceptional bedside manner... most of the med students I started school with are so burnt-out now that they don't give a fuck about almost any of their patients (other than "don't get sued").