4 ms·

> ... it means that you are probably better off with GPT than with a real doctor if you don't have something super peculiar that would take even true experts a

by pintxo 2y ago

> ... it means that you are probably better off with GPT than with a real doctor if you don't have something super peculiar that would take even true experts a lot of close attention and research to figure out. I would argue the opposite. Any reasonably well trained AI should outperform doctors on rare/exotic things, that most people will see never or only every other year. As it will still be described in the literature, the AI will have picked it up, but our average doctor is more likely than not to have forgotten about it since med-school, if she ever heard about it in the first place. But when it comes to the normal day-2-day stuff, I am more afraid of the AI having a sudden case of hallucinations vs the doctor who probably has seen that same case yesterday and might not apply the latest state-of-the-art treatment [1], but it will probably be fine. [1] It often takes 5 years or more for the medical state-of-the-art to transpire into daily practice "in the field". Edit: Add citation and fix typo.

sigmoid10 2y ago
>vs the doctor who probably has seen that same case yesterday This is actually super dangerous and probably the cause for a lot of misdiagnosing and medical errors these days. From the paper, it appears that GPT does much better in these cases.
- rscho 2y ago
  This study does not test the doc's 'feel'. That's how an experienced doc evaluates patients, in good part. You develop an unconscious 'feel' for the stuff you see often. So, calling this study a fair comparison is actually pretty laughable.
  - sigmoid10 2y ago
    This "feel" is just experience - aka having seen many things without remembering all the circumstances. It is nothing supernatural.
    - rscho 2y ago
      Well, yes that's my point. You deprive the docs of most stimuli they really rely on, and call it a comparative study. It's like saying A* is better at finding its way into a maze than a blindfolded man, and call it success.
      - pintxo 2y ago
        Which is how lots of research, including medical/pharmaceutical [1] studies, are done. [1] Sure, you have to test your new cancer drug on actual cancer patients, but why include the 65 year old obese patient, when you could instead include the 32 year old triathlon athlete into you cohort?
      - sigmoid10 2y ago
        >like saying A* is better at finding its way into a maze than a blindfolded man It's not just better than a blind man. It's better than any human, once you scale the task up beyond a certain point. Consequently, we should not get hung up on the path taken if the ultimate goal is the best possible care for patients in our overloaded public systems. And this paper suggests there is great potential.
        rscho 2y ago
        Of course there is potential. There is no doubt that computers and robots will someday outperform docs. But to me (an experienced anesthesiologist), this day is probably futher off than most believe. Especially the robot part.
        pintxo 2y ago
        Guess the point of this sub-threat is, that as long as you cannot show it in a well-done study, the potential is just that: potential. But not something tangible.
        rscho 2y ago
        The big problem is: what does a well-conducted study mean to people? For me personally, I won't be convinced until machines get the same info as the doc, meaning the whole messy file raw plus what can be heard, seen or touched directly, and that's it!
  - roenxi 2y ago
    I don't think we want the doctor doing that though, that is where a lot of bias and inaccuracy creeps into the system. It'd be better if they used a checklist and evidence based process that can be engineered and improved over time. If there is some evidence they are gathering that ChatGPT doesn't have access to, an effort should be made to capture and systematise it. But there probably isn't very much of that for a typical doctors visit.
    - rscho 2y ago
      > I don't think we want the doctor doing that You don't want your doc to give your problem his/her best shot !? > used a checklist and evidence based process that can be engineered and improved over time. Enticing and logically evident, isn't it? Well, except in practice it would cost so much to do that, that it's clearly not economically doable. And I'm not even sure it's technically doable in 2024. > If there is some evidence they are gathering that ChatGPT doesn't have access to, an effort should be made to capture and systematise it. But there probably isn't very much of that for a typical doctors visit. GPT is missing most of the info, because it's not equipped with 5 human senses and a brain to aggregate them. And that's, by far the most important info. You can see the challenge in solving this problem at a large scale, I'm sure.
      - roenxi 2y ago
        > You don't want your doc to give your problem his/her best shot !? I don't think the doctors gut feel is going to be their best shot. I want them using logic, evidence and reason. The medical industry has a long history of people slowly managing to convince doctors to use evidence instead of making bad guesses. Explicit processes generally lead to far superior outcomes.
        rscho 2y ago
        Bluntly put, you are wrong regarding today's medicine. In 2024, medicine is a heterogeneous mix of art and science. Even for experts, separating real science from what others would want you to believe is difficult, and experience is essential in most cases to bridge the gaps in knowledge. The best summary I can formulate is 'in 2024, a good doc is a stellar educated guesser relying on lots of inexact knowledge, a substantial amount of experience, and a sprinkle of logical reasoning in the rare but critical steps where reasoning is applicable'.

by pintxo 2y ago

sigmoid10 2y ago

>vs the doctor who probably has seen that same case yesterday This is actually super dangerous and probably the cause for a lot of misdiagnosing and medical errors these days. From the paper, it appears that GPT does much better in these cases.

rscho 2y ago
This study does not test the doc's 'feel'. That's how an experienced doc evaluates patients, in good part. You develop an unconscious 'feel' for the stuff you see often. So, calling this study a fair comparison is actually pretty laughable.
- sigmoid10 2y ago
  This "feel" is just experience - aka having seen many things without remembering all the circumstances. It is nothing supernatural.
  - rscho 2y ago
    Well, yes that's my point. You deprive the docs of most stimuli they really rely on, and call it a comparative study. It's like saying A* is better at finding its way into a maze than a blindfolded man, and call it success.
    - pintxo 2y ago
      Which is how lots of research, including medical/pharmaceutical [1] studies, are done. [1] Sure, you have to test your new cancer drug on actual cancer patients, but why include the 65 year old obese patient, when you could instead include the 32 year old triathlon athlete into you cohort?
    - sigmoid10 2y ago
      >like saying A* is better at finding its way into a maze than a blindfolded man It's not just better than a blind man. It's better than any human, once you scale the task up beyond a certain point. Consequently, we should not get hung up on the path taken if the ultimate goal is the best possible care for patients in our overloaded public systems. And this paper suggests there is great potential.
      - rscho 2y ago
        Of course there is potential. There is no doubt that computers and robots will someday outperform docs. But to me (an experienced anesthesiologist), this day is probably futher off than most believe. Especially the robot part.
      - pintxo 2y ago
        Guess the point of this sub-threat is, that as long as you cannot show it in a well-done study, the potential is just that: potential. But not something tangible.
        rscho 2y ago
        The big problem is: what does a well-conducted study mean to people? For me personally, I won't be convinced until machines get the same info as the doc, meaning the whole messy file raw plus what can be heard, seen or touched directly, and that's it!
- roenxi 2y ago
  I don't think we want the doctor doing that though, that is where a lot of bias and inaccuracy creeps into the system. It'd be better if they used a checklist and evidence based process that can be engineered and improved over time. If there is some evidence they are gathering that ChatGPT doesn't have access to, an effort should be made to capture and systematise it. But there probably isn't very much of that for a typical doctors visit.
  - rscho 2y ago
    > I don't think we want the doctor doing that You don't want your doc to give your problem his/her best shot !? > used a checklist and evidence based process that can be engineered and improved over time. Enticing and logically evident, isn't it? Well, except in practice it would cost so much to do that, that it's clearly not economically doable. And I'm not even sure it's technically doable in 2024. > If there is some evidence they are gathering that ChatGPT doesn't have access to, an effort should be made to capture and systematise it. But there probably isn't very much of that for a typical doctors visit. GPT is missing most of the info, because it's not equipped with 5 human senses and a brain to aggregate them. And that's, by far the most important info. You can see the challenge in solving this problem at a large scale, I'm sure.
    - roenxi 2y ago
      > You don't want your doc to give your problem his/her best shot !? I don't think the doctors gut feel is going to be their best shot. I want them using logic, evidence and reason. The medical industry has a long history of people slowly managing to convince doctors to use evidence instead of making bad guesses. Explicit processes generally lead to far superior outcomes.
      - rscho 2y ago
        Bluntly put, you are wrong regarding today's medicine. In 2024, medicine is a heterogeneous mix of art and science. Even for experts, separating real science from what others would want you to believe is difficult, and experience is essential in most cases to bridge the gaps in knowledge. The best summary I can formulate is 'in 2024, a good doc is a stellar educated guesser relying on lots of inexact knowledge, a substantial amount of experience, and a sprinkle of logical reasoning in the rare but critical steps where reasoning is applicable'.