5 ms·

One med student and nine physicians aren't the greatest study dataset for comparing human performance, but the results are still pretty wild. GPT4-V can reliabl

One med student and nine physicians aren't the greatest study dataset for comparing human performance, but the results are still pretty wild. GPT4-V can reliably outperform or match doctors on all but the hardest questions in a closed book setting. If you take their results at face value and apply them to modern medical care environments, it means that you are probably better off with GPT than with a real doctor if you don't have something super peculiar that would take even true experts a lot of close attention and research to figure out. It will certainly beat an uninterested or overworked clinician. I'm really looking forward to studies that look at performance comparisons in realistic environments. I believe there is a potential revolution brewing for PCPs. Having someone who actually listens and has real in-depth knowledge that goes beyond normal med students could be a game changer when compared to the usual conveyor belt care that gives everyone the same diagnosis based on a quick glance.

rscho 2y ago
> better off with GPT than with a real doctor if you don't have something super peculiar ... or if you've got anything that requires know-how/practical experience, which coincidentally is what makes most of medicine.
- sigmoid10 2y ago
  That is not what this study suggests.
  - rscho 2y ago
    This was not tested in the study.
    - sigmoid10 2y ago
      It was. Look at the dataset. This is stuff that takes tons of experience to work with and it also explains why GPT was so much better than the med student. Although my original N=1 criticism here still applies.
      - rscho 2y ago
        We're not speaking of the same thing when we say 'know-how', then. No, it was not tested.
        sigmoid10 2y ago
        If you're referring to physical examinations (and I don't know what else you could be talking about since you didn't specify it), then the point still stands. You could theoretically make an argument that the sensation of performing the examination yourself vs. just hearing a first-hand account could make a difference, and I would agree. But in a clinical setting you won't have the expert chief physician re-do the physical either, so the argument is kind of pointless.
        rscho 2y ago
        >performing the examination yourself vs. just hearing a first-hand account could make a difference That makes a world of difference. It's actually where most of a doc's performance resides. > in a clinical setting you won't have the expert chief physician re-do the physical either And you don't need it. Lots of thousands of fellows or even some advanced resident are already at the approriate level of experience. Actually, fellows are often superior because they've got a superior volume of practice.
        sigmoid10 2y ago
        >That makes a world of difference Got any source on that? In the paper they certainly don't assume that. >Lots of thousands of fellows or even some advanced resident Same argument as before. We simply don't have enough of those in western medical care due to aging societies.
        jstummbillig 2y ago
        Share what you mean?
pintxo 2y ago
> ... it means that you are probably better off with GPT than with a real doctor if you don't have something super peculiar that would take even true experts a lot of close attention and research to figure out. I would argue the opposite. Any reasonably well trained AI should outperform doctors on rare/exotic things, that most people will see never or only every other year. As it will still be described in the literature, the AI will have picked it up, but our average doctor is more likely than not to have forgotten about it since med-school, if she ever heard about it in the first place. But when it comes to the normal day-2-day stuff, I am more afraid of the AI having a sudden case of hallucinations vs the doctor who probably has seen that same case yesterday and might not apply the latest state-of-the-art treatment [1], but it will probably be fine. [1] It often takes 5 years or more for the medical state-of-the-art to transpire into daily practice "in the field". Edit: Add citation and fix typo.
- sigmoid10 2y ago
  >vs the doctor who probably has seen that same case yesterday This is actually super dangerous and probably the cause for a lot of misdiagnosing and medical errors these days. From the paper, it appears that GPT does much better in these cases.
  - rscho 2y ago
    This study does not test the doc's 'feel'. That's how an experienced doc evaluates patients, in good part. You develop an unconscious 'feel' for the stuff you see often. So, calling this study a fair comparison is actually pretty laughable.
    - sigmoid10 2y ago
      This "feel" is just experience - aka having seen many things without remembering all the circumstances. It is nothing supernatural.
      - rscho 2y ago
        Well, yes that's my point. You deprive the docs of most stimuli they really rely on, and call it a comparative study. It's like saying A* is better at finding its way into a maze than a blindfolded man, and call it success.
lucianbr 2y ago
> if you don't have something super peculiar How would I know beforehand if I have something fit for an AI or for a human?
- EForEndeavour 2y ago
  Peculiar issues are by definition less common than the usual issues, so you're more likely to have a common issue than a peculiar one.
  - lucianbr 2y ago
    How does that help? I still don't know ahead of time if I have a common issue or an uncommon one, to choose the right expert. Are you saying just always assume you have a common, non-peculiar issue, and if I happen to have the uncommon one, bad luck, got the wrong treatment? At least take solace in the fact that it doesn't happen often?
taneq 2y ago
> It will certainly beat an uninterested or overworked clinician. This is what most discussions seem to disregard. Sure, a celebrated expert on a good day with prep time may still outperform a machine learning system. They will also outperform a mediocre practitioner at the end of a long week, probably by a larger margin.

rscho 2y ago

> better off with GPT than with a real doctor if you don't have something super peculiar ... or if you've got anything that requires know-how/practical experience, which coincidentally is what makes most of medicine.

sigmoid10 2y ago
That is not what this study suggests.
- rscho 2y ago
  This was not tested in the study.
  - sigmoid10 2y ago
    It was. Look at the dataset. This is stuff that takes tons of experience to work with and it also explains why GPT was so much better than the med student. Although my original N=1 criticism here still applies.
    - rscho 2y ago
      We're not speaking of the same thing when we say 'know-how', then. No, it was not tested.
      - sigmoid10 2y ago
        If you're referring to physical examinations (and I don't know what else you could be talking about since you didn't specify it), then the point still stands. You could theoretically make an argument that the sensation of performing the examination yourself vs. just hearing a first-hand account could make a difference, and I would agree. But in a clinical setting you won't have the expert chief physician re-do the physical either, so the argument is kind of pointless.
        rscho 2y ago
        >performing the examination yourself vs. just hearing a first-hand account could make a difference That makes a world of difference. It's actually where most of a doc's performance resides. > in a clinical setting you won't have the expert chief physician re-do the physical either And you don't need it. Lots of thousands of fellows or even some advanced resident are already at the approriate level of experience. Actually, fellows are often superior because they've got a superior volume of practice.
        sigmoid10 2y ago
        >That makes a world of difference Got any source on that? In the paper they certainly don't assume that. >Lots of thousands of fellows or even some advanced resident Same argument as before. We simply don't have enough of those in western medical care due to aging societies.
      - jstummbillig 2y ago
        Share what you mean?

pintxo 2y ago

> ... it means that you are probably better off with GPT than with a real doctor if you don't have something super peculiar that would take even true experts a lot of close attention and research to figure out. I would argue the opposite. Any reasonably well trained AI should outperform doctors on rare/exotic things, that most people will see never or only every other year. As it will still be described in the literature, the AI will have picked it up, but our average doctor is more likely than not to have forgotten about it since med-school, if she ever heard about it in the first place. But when it comes to the normal day-2-day stuff, I am more afraid of the AI having a sudden case of hallucinations vs the doctor who probably has seen that same case yesterday and might not apply the latest state-of-the-art treatment [1], but it will probably be fine. [1] It often takes 5 years or more for the medical state-of-the-art to transpire into daily practice "in the field". Edit: Add citation and fix typo.

sigmoid10 2y ago
>vs the doctor who probably has seen that same case yesterday This is actually super dangerous and probably the cause for a lot of misdiagnosing and medical errors these days. From the paper, it appears that GPT does much better in these cases.
- rscho 2y ago
  This study does not test the doc's 'feel'. That's how an experienced doc evaluates patients, in good part. You develop an unconscious 'feel' for the stuff you see often. So, calling this study a fair comparison is actually pretty laughable.
  - sigmoid10 2y ago
    This "feel" is just experience - aka having seen many things without remembering all the circumstances. It is nothing supernatural.
    - rscho 2y ago
      Well, yes that's my point. You deprive the docs of most stimuli they really rely on, and call it a comparative study. It's like saying A* is better at finding its way into a maze than a blindfolded man, and call it success.

lucianbr 2y ago

> if you don't have something super peculiar How would I know beforehand if I have something fit for an AI or for a human?

EForEndeavour 2y ago
Peculiar issues are by definition less common than the usual issues, so you're more likely to have a common issue than a peculiar one.
- lucianbr 2y ago
  How does that help? I still don't know ahead of time if I have a common issue or an uncommon one, to choose the right expert. Are you saying just always assume you have a common, non-peculiar issue, and if I happen to have the uncommon one, bad luck, got the wrong treatment? At least take solace in the fact that it doesn't happen often?

taneq 2y ago

> It will certainly beat an uninterested or overworked clinician. This is what most discussions seem to disregard. Sure, a celebrated expert on a good day with prep time may still outperform a machine learning system. They will also outperform a mediocre practitioner at the end of a long week, probably by a larger margin.