3 ms·

This study does not test the doc's 'feel'. That's how an experienced doc evaluates patients, in good part. You develop an unconscious 'feel' for the stuff you s

by rscho 2y ago

This study does not test the doc's 'feel'. That's how an experienced doc evaluates patients, in good part. You develop an unconscious 'feel' for the stuff you see often. So, calling this study a fair comparison is actually pretty laughable.

sigmoid10 2y ago
This "feel" is just experience - aka having seen many things without remembering all the circumstances. It is nothing supernatural.
- rscho 2y ago
  Well, yes that's my point. You deprive the docs of most stimuli they really rely on, and call it a comparative study. It's like saying A* is better at finding its way into a maze than a blindfolded man, and call it success.
  - pintxo 2y ago
    Which is how lots of research, including medical/pharmaceutical [1] studies, are done. [1] Sure, you have to test your new cancer drug on actual cancer patients, but why include the 65 year old obese patient, when you could instead include the 32 year old triathlon athlete into you cohort?
  - sigmoid10 2y ago
    >like saying A* is better at finding its way into a maze than a blindfolded man It's not just better than a blind man. It's better than any human, once you scale the task up beyond a certain point. Consequently, we should not get hung up on the path taken if the ultimate goal is the best possible care for patients in our overloaded public systems. And this paper suggests there is great potential.
    - rscho 2y ago
      Of course there is potential. There is no doubt that computers and robots will someday outperform docs. But to me (an experienced anesthesiologist), this day is probably futher off than most believe. Especially the robot part.
    - pintxo 2y ago
      Guess the point of this sub-threat is, that as long as you cannot show it in a well-done study, the potential is just that: potential. But not something tangible.
      - rscho 2y ago
        The big problem is: what does a well-conducted study mean to people? For me personally, I won't be convinced until machines get the same info as the doc, meaning the whole messy file raw plus what can be heard, seen or touched directly, and that's it!
        pintxo 2y ago
        If a machine get's (statistically significant) better results than the docs with less information, that should be good as well, no? Although I would tend to say we should expect machines to be significantly better than the average doctor. Before we role them out to everyone we should require them to be at least as good as the top 10% of doctors, I guess.
      - sigmoid10 2y ago
        I.e. literally the first sentence if the top post.
  - Mathnerd314 2y ago
    I'm not saying doctor's expertise isn't valuable, but studies with even simple linear models have shown that it is much better to feed a doctor's expertise into an expert AI system than to try to teach the AI system to the doctor. And the weighting of the expertise relative to other factors is often lower than you might expect. Even a simple yes/no checklist can be invaluable if it is designed by the top minds of the field.
    - rscho 2y ago
      Clinical scoring (which is what I assume you are referring to as linear models) is mostly useful when the diagnosis is already more or less identified, to estimate prognosis and choose treatment options. They're no exploratory tools. When you 'feed the doc's expertise into a linear system', it means the expertise required to identify the problem in the first place has already done its job.
      - Mathnerd314 2y ago
        I don't have the original reference, it was an invited talk at UChicago, but something like https://www.sciencedirect.com/science/article/abs/pii/S0957417408001851 https://www.sciencedirect.com/science/article/abs/pii/S09574... for example. But regarding diagnosis, there are symptom checkers where you feed stuff in. I get the impression that multiple companies are in the process of building such tools based on crawling research papers to diagnose even very rare conditions. IIRC IBM Watson had such functionality.
        toomuchtodo 2y ago
        NHS should be building this for PubMed and offering it for free.
        pintxo 2y ago
        One example: https://ada.com https://ada.com I mean, it's not like rocket science. More like investing the effort to build a really big decision tree.
roenxi 2y ago
I don't think we want the doctor doing that though, that is where a lot of bias and inaccuracy creeps into the system. It'd be better if they used a checklist and evidence based process that can be engineered and improved over time. If there is some evidence they are gathering that ChatGPT doesn't have access to, an effort should be made to capture and systematise it. But there probably isn't very much of that for a typical doctors visit.
- rscho 2y ago
  > I don't think we want the doctor doing that You don't want your doc to give your problem his/her best shot !? > used a checklist and evidence based process that can be engineered and improved over time. Enticing and logically evident, isn't it? Well, except in practice it would cost so much to do that, that it's clearly not economically doable. And I'm not even sure it's technically doable in 2024. > If there is some evidence they are gathering that ChatGPT doesn't have access to, an effort should be made to capture and systematise it. But there probably isn't very much of that for a typical doctors visit. GPT is missing most of the info, because it's not equipped with 5 human senses and a brain to aggregate them. And that's, by far the most important info. You can see the challenge in solving this problem at a large scale, I'm sure.
  - roenxi 2y ago
    > You don't want your doc to give your problem his/her best shot !? I don't think the doctors gut feel is going to be their best shot. I want them using logic, evidence and reason. The medical industry has a long history of people slowly managing to convince doctors to use evidence instead of making bad guesses. Explicit processes generally lead to far superior outcomes.
    - rscho 2y ago
      Bluntly put, you are wrong regarding today's medicine. In 2024, medicine is a heterogeneous mix of art and science. Even for experts, separating real science from what others would want you to believe is difficult, and experience is essential in most cases to bridge the gaps in knowledge. The best summary I can formulate is 'in 2024, a good doc is a stellar educated guesser relying on lots of inexact knowledge, a substantial amount of experience, and a sprinkle of logical reasoning in the rare but critical steps where reasoning is applicable'.

by rscho 2y ago

sigmoid10 2y ago

This "feel" is just experience - aka having seen many things without remembering all the circumstances. It is nothing supernatural.

rscho 2y ago
Well, yes that's my point. You deprive the docs of most stimuli they really rely on, and call it a comparative study. It's like saying A* is better at finding its way into a maze than a blindfolded man, and call it success.
- pintxo 2y ago
  Which is how lots of research, including medical/pharmaceutical [1] studies, are done. [1] Sure, you have to test your new cancer drug on actual cancer patients, but why include the 65 year old obese patient, when you could instead include the 32 year old triathlon athlete into you cohort?
- sigmoid10 2y ago
  >like saying A* is better at finding its way into a maze than a blindfolded man It's not just better than a blind man. It's better than any human, once you scale the task up beyond a certain point. Consequently, we should not get hung up on the path taken if the ultimate goal is the best possible care for patients in our overloaded public systems. And this paper suggests there is great potential.
  - rscho 2y ago
    Of course there is potential. There is no doubt that computers and robots will someday outperform docs. But to me (an experienced anesthesiologist), this day is probably futher off than most believe. Especially the robot part.
  - pintxo 2y ago
    Guess the point of this sub-threat is, that as long as you cannot show it in a well-done study, the potential is just that: potential. But not something tangible.
    - rscho 2y ago
      The big problem is: what does a well-conducted study mean to people? For me personally, I won't be convinced until machines get the same info as the doc, meaning the whole messy file raw plus what can be heard, seen or touched directly, and that's it!
      - pintxo 2y ago
        If a machine get's (statistically significant) better results than the docs with less information, that should be good as well, no? Although I would tend to say we should expect machines to be significantly better than the average doctor. Before we role them out to everyone we should require them to be at least as good as the top 10% of doctors, I guess.
    - sigmoid10 2y ago
      I.e. literally the first sentence if the top post.
- Mathnerd314 2y ago
  I'm not saying doctor's expertise isn't valuable, but studies with even simple linear models have shown that it is much better to feed a doctor's expertise into an expert AI system than to try to teach the AI system to the doctor. And the weighting of the expertise relative to other factors is often lower than you might expect. Even a simple yes/no checklist can be invaluable if it is designed by the top minds of the field.
  - rscho 2y ago
    Clinical scoring (which is what I assume you are referring to as linear models) is mostly useful when the diagnosis is already more or less identified, to estimate prognosis and choose treatment options. They're no exploratory tools. When you 'feed the doc's expertise into a linear system', it means the expertise required to identify the problem in the first place has already done its job.
    - Mathnerd314 2y ago
      I don't have the original reference, it was an invited talk at UChicago, but something like https://www.sciencedirect.com/science/article/abs/pii/S0957417408001851 https://www.sciencedirect.com/science/article/abs/pii/S09574... for example. But regarding diagnosis, there are symptom checkers where you feed stuff in. I get the impression that multiple companies are in the process of building such tools based on crawling research papers to diagnose even very rare conditions. IIRC IBM Watson had such functionality.
      - toomuchtodo 2y ago
        NHS should be building this for PubMed and offering it for free.
      - pintxo 2y ago
        One example: https://ada.com https://ada.com I mean, it's not like rocket science. More like investing the effort to build a really big decision tree.

roenxi 2y ago

I don't think we want the doctor doing that though, that is where a lot of bias and inaccuracy creeps into the system. It'd be better if they used a checklist and evidence based process that can be engineered and improved over time. If there is some evidence they are gathering that ChatGPT doesn't have access to, an effort should be made to capture and systematise it. But there probably isn't very much of that for a typical doctors visit.

rscho 2y ago
> I don't think we want the doctor doing that You don't want your doc to give your problem his/her best shot !? > used a checklist and evidence based process that can be engineered and improved over time. Enticing and logically evident, isn't it? Well, except in practice it would cost so much to do that, that it's clearly not economically doable. And I'm not even sure it's technically doable in 2024. > If there is some evidence they are gathering that ChatGPT doesn't have access to, an effort should be made to capture and systematise it. But there probably isn't very much of that for a typical doctors visit. GPT is missing most of the info, because it's not equipped with 5 human senses and a brain to aggregate them. And that's, by far the most important info. You can see the challenge in solving this problem at a large scale, I'm sure.
- roenxi 2y ago
  > You don't want your doc to give your problem his/her best shot !? I don't think the doctors gut feel is going to be their best shot. I want them using logic, evidence and reason. The medical industry has a long history of people slowly managing to convince doctors to use evidence instead of making bad guesses. Explicit processes generally lead to far superior outcomes.
  - rscho 2y ago
    Bluntly put, you are wrong regarding today's medicine. In 2024, medicine is a heterogeneous mix of art and science. Even for experts, separating real science from what others would want you to believe is difficult, and experience is essential in most cases to bridge the gaps in knowledge. The best summary I can formulate is 'in 2024, a good doc is a stellar educated guesser relying on lots of inexact knowledge, a substantial amount of experience, and a sprinkle of logical reasoning in the rare but critical steps where reasoning is applicable'.