5 ms·

ChatGPT Has Passed the United States Medical Licensing Examination (USMLE)

45 points by zekrioca 4y ago

jedberg 4y ago
I'm not sure if this is more of a pro in the ChatGPT column or an indictment of the USMLE. Although I don't think the USMLE is known as an easy exam, so I guess ChatGPT really is that good now?
- SketchySeaBeast 4y ago
  Isn't this yet another case of "you have to memorize and regurgitate a ton of information" which people are bad at but something like ChatGPT is exceptionally good at?
  - swagonomixxx 4y ago
    I think this is one point that I don't see being made often enough. ChatGPT is exceptional at retrieval of information, but if it starts passing tests like the MCAT and LSAT and so on, I feel this is more that our testing is just crap, rather than ChatGPT being "smarter" than humans.
    - jfengel 4y ago
      Being a doctor does require a lot of just plain information retrieval. Human bodies have a kajillion parts interacting in a kajillion-squared ways. And we usually don't get to look directly at most of the parts; we have to infer what's going on from very indirect evidence. This isn't one of those "memorize nothing you can look up" situations. You just have to have it at your disposal before you can do actual medicine. There may be a day when a computer can do most of the memorization scutwork and you can get back to the real work, but we're not there yet. It's not the test's fault that it's testing medicine as it's practiced today.
- thethirdone 4y ago
  > In 2020, 98% of DO and MD students passed the exam on their first try. But if you don’t, you are given four attempts to pass each different level of the USMLE exam. If you fail, you receive a score report that offers feedback on the areas in which you struggled. from https://www.ama-assn.org/medical-students/usmle-step-1-2/usmle-step-1-score-switch-answers-medical-students-top-questions https://www.ama-assn.org/medical-students/usmle-step-1-2/usm... It would seem USMLE is not hard to pass.
transitivebs 4y ago
Does anyone know the methodology? E.g., what were the inputs and context windows given to ChatGPT?
- maxerickson 4y ago
  The full text is available and describes how they approached it. I didn't look to see if the actual inputs are in there.
  - Incipient 4y ago
    It's available on an "online data system" says the article. But no where do they say what the damn system is.
Slighted 4y ago
Just because it can pass the test doesn't mean that its behavior will be consistently correct. Trusting ChatGPT in medical applications is the same as trusting it in programming applications: It will work for simple to intermediate tasks and may even look as though its explaining itself properly, but for specialized tasks, sufficiently complicated ones, and even in simple tasks, it can't be expected to product correct results consistently.
- mshake2 4y ago
  >Just because it can pass the test doesn't mean that its behavior will be consistently correct. Now do doctors.
- Gigachad 4y ago
  The main observation I've had of these AI tools is they don't learn the same way people do. You see the art generation ones which are able to do hyper realistic texturing and shading, but fail to get the right number of fingers on a hand. In the say way ChatGPT is able to answer some extremely tricky questions right while failing trivial ones.
  - haimez 4y ago
    They don’t even learn, they confabulate. “Correct” responses are an accident, and there is no notion of (especially medical) correctness involved.
- xiphias2 4y ago
  I already started using it for medical applications. My friend had allergic reaction to Monster Energy drink today, and while I could find a lot of documents in Google, ChatGPT explained what I need to know, what medicine she has to buy (antihistamine, it worked), when is it important to call ambulance, what to take. I can Google all these things, but all the SEO spam pages that I get just take a lot of time and effort. Also she doesn't have money for private doctor, and to get to talk to a doctor takes weeks. She was also at the emergency last time when she had the reaction, and they made her wait 8 hours there :( ChatGPT just works for medical problems (but I still double check the answers)
xiphias2 4y ago
,, The most recent iteration of the GPT LLM (GPT3) achieved 46% accuracy with zero prompting12, which marginally improved to 50% with further model training. Previous models, merely months prior, performed at 36.7%13. In this present study, ChatGPT performed at >50% accuracy across all examinations, exceeding 60% in most analyses'' I have heared many times that ChatGPT is basically GPT3, but somehow it feels like it gives more correct answers. Now there's data to show it.
YeGoblynQueenne 4y ago
The HN title is misleading. The study is titled: Performance of ChatGPT on USMLE: Potential for AI-Assisted Medical Education Using Large Language Models The study did not attempt to get ChatGPT to pass any exam. Rather, ChatGPT was question on material in past exams. The authors are enthusiastic about ChatGPT's performance but nowhere do they claim it "passed the US Medical Licensing Exam" as the title claims. Indeed, the study concludes that ChatGPT's performance is less than enough to pass the exam: ChatGPT yields moderate accuracy approaching passing performance on USMLE (line 287 of the manuscript). @zekrioca, please correct the title so that it more closely reflects the results of the linked study.
- zekrioca 4y ago
  Sorry you and all about the misunderstanding, I’d gladly fix it but I guess it is late? I couldn’t get the real title to fit within the space limit, and improvised from the abstract.
  - YeGoblynQueenne 4y ago
    Hey, no worries, thanks for posting.
tokai 4y ago
The hn title is really pushing way beyond what the study can shoulder. The title from the result chapter sums it up better; "ChatGPT yields moderate accuracy approaching passing performance on USMLE" The study was done on public available questions that have been used in usmle exams. They used two physician reviewers to judge ChatGPTs textual answers. While really interesting no exam was actually passed, and the authors never claim that.
- option 4y ago
  Public questions really means they have likely been in training data together with their answers
  - tokai 4y ago
    From the paper: Random spot checking was performed to ensure that none of the answers, explanations, or related content were indexed on Google prior to January 1, 2022, representing the last date accessible to the ChatGPT training dataset. So its maybe not a problem, who knows
simonhaidarjh 4y ago
Heart palpitations
eig 4y ago
Medical student here. I’m impressed by ChatGPT’s ability to come close to passing Step 2/3. I tried plugging in my own question banks that I used for studying and it got many right. Even when it was wrong it was also “almost right” and its answer would not be disparaged on rounds. However one thing to keep in mind: a bot passing these exams alone will not disrupt medicine. There are plenty of doctors today that could ace these exams. I don’t think that the ability to outsource medical decision making to a bot will change much, especially since 90% of medical decision making is already outsourced to a flowchart or an “expert attending” physician. Much of the value of doctors today comes from procedures, patient interaction, legal liability, and patient trust, which cannot yet be done by a bot.

45 points by zekrioca 4y ago

I'm not sure if this is more of a pro in the ChatGPT column or an indictment of the USMLE. Although I don't think the USMLE is known as an easy exam, so I guess ChatGPT really is that good now?

SketchySeaBeast 4y ago
Isn't this yet another case of "you have to memorize and regurgitate a ton of information" which people are bad at but something like ChatGPT is exceptionally good at?
- swagonomixxx 4y ago
  I think this is one point that I don't see being made often enough. ChatGPT is exceptional at retrieval of information, but if it starts passing tests like the MCAT and LSAT and so on, I feel this is more that our testing is just crap, rather than ChatGPT being "smarter" than humans.
  - jfengel 4y ago
    Being a doctor does require a lot of just plain information retrieval. Human bodies have a kajillion parts interacting in a kajillion-squared ways. And we usually don't get to look directly at most of the parts; we have to infer what's going on from very indirect evidence. This isn't one of those "memorize nothing you can look up" situations. You just have to have it at your disposal before you can do actual medicine. There may be a day when a computer can do most of the memorization scutwork and you can get back to the real work, but we're not there yet. It's not the test's fault that it's testing medicine as it's practiced today.
thethirdone 4y ago
> In 2020, 98% of DO and MD students passed the exam on their first try. But if you don’t, you are given four attempts to pass each different level of the USMLE exam. If you fail, you receive a score report that offers feedback on the areas in which you struggled. from https://www.ama-assn.org/medical-students/usmle-step-1-2/usmle-step-1-score-switch-answers-medical-students-top-questions https://www.ama-assn.org/medical-students/usmle-step-1-2/usm... It would seem USMLE is not hard to pass.

transitivebs 4y ago

Does anyone know the methodology? E.g., what were the inputs and context windows given to ChatGPT?

maxerickson 4y ago
The full text is available and describes how they approached it. I didn't look to see if the actual inputs are in there.
- Incipient 4y ago
  It's available on an "online data system" says the article. But no where do they say what the damn system is.

Slighted 4y ago

Just because it can pass the test doesn't mean that its behavior will be consistently correct. Trusting ChatGPT in medical applications is the same as trusting it in programming applications: It will work for simple to intermediate tasks and may even look as though its explaining itself properly, but for specialized tasks, sufficiently complicated ones, and even in simple tasks, it can't be expected to product correct results consistently.

mshake2 4y ago
>Just because it can pass the test doesn't mean that its behavior will be consistently correct. Now do doctors.
Gigachad 4y ago
The main observation I've had of these AI tools is they don't learn the same way people do. You see the art generation ones which are able to do hyper realistic texturing and shading, but fail to get the right number of fingers on a hand. In the say way ChatGPT is able to answer some extremely tricky questions right while failing trivial ones.
- haimez 4y ago
  They don’t even learn, they confabulate. “Correct” responses are an accident, and there is no notion of (especially medical) correctness involved.
xiphias2 4y ago
I already started using it for medical applications. My friend had allergic reaction to Monster Energy drink today, and while I could find a lot of documents in Google, ChatGPT explained what I need to know, what medicine she has to buy (antihistamine, it worked), when is it important to call ambulance, what to take. I can Google all these things, but all the SEO spam pages that I get just take a lot of time and effort. Also she doesn't have money for private doctor, and to get to talk to a doctor takes weeks. She was also at the emergency last time when she had the reaction, and they made her wait 8 hours there :( ChatGPT just works for medical problems (but I still double check the answers)

xiphias2 4y ago

,, The most recent iteration of the GPT LLM (GPT3) achieved 46% accuracy with zero prompting12, which marginally improved to 50% with further model training. Previous models, merely months prior, performed at 36.7%13. In this present study, ChatGPT performed at >50% accuracy across all examinations, exceeding 60% in most analyses'' I have heared many times that ChatGPT is basically GPT3, but somehow it feels like it gives more correct answers. Now there's data to show it.

YeGoblynQueenne 4y ago

The HN title is misleading. The study is titled: Performance of ChatGPT on USMLE: Potential for AI-Assisted Medical Education Using Large Language Models The study did not attempt to get ChatGPT to pass any exam. Rather, ChatGPT was question on material in past exams. The authors are enthusiastic about ChatGPT's performance but nowhere do they claim it "passed the US Medical Licensing Exam" as the title claims. Indeed, the study concludes that ChatGPT's performance is less than enough to pass the exam: ChatGPT yields moderate accuracy approaching passing performance on USMLE (line 287 of the manuscript). @zekrioca, please correct the title so that it more closely reflects the results of the linked study.

zekrioca 4y ago
Sorry you and all about the misunderstanding, I’d gladly fix it but I guess it is late? I couldn’t get the real title to fit within the space limit, and improvised from the abstract.
- YeGoblynQueenne 4y ago
  Hey, no worries, thanks for posting.

tokai 4y ago

The hn title is really pushing way beyond what the study can shoulder. The title from the result chapter sums it up better; "ChatGPT yields moderate accuracy approaching passing performance on USMLE" The study was done on public available questions that have been used in usmle exams. They used two physician reviewers to judge ChatGPTs textual answers. While really interesting no exam was actually passed, and the authors never claim that.

option 4y ago
Public questions really means they have likely been in training data together with their answers
- tokai 4y ago
  From the paper: Random spot checking was performed to ensure that none of the answers, explanations, or related content were indexed on Google prior to January 1, 2022, representing the last date accessible to the ChatGPT training dataset. So its maybe not a problem, who knows

simonhaidarjh 4y ago

Heart palpitations

eig 4y ago

Medical student here. I’m impressed by ChatGPT’s ability to come close to passing Step 2/3. I tried plugging in my own question banks that I used for studying and it got many right. Even when it was wrong it was also “almost right” and its answer would not be disparaged on rounds. However one thing to keep in mind: a bot passing these exams alone will not disrupt medicine. There are plenty of doctors today that could ace these exams. I don’t think that the ability to outsource medical decision making to a bot will change much, especially since 90% of medical decision making is already outsourced to a flowchart or an “expert attending” physician. Much of the value of doctors today comes from procedures, patient interaction, legal liability, and patient trust, which cannot yet be done by a bot.