5 ms·

Unpopular take: ChatGPT is better at diagnosing medical problems than a doctor who does a cursory evaluation. By cursory, I mean an evaluation that does not ta

Unpopular take: ChatGPT is better at diagnosing medical problems than a doctor who does a cursory evaluation. By cursory, I mean an evaluation that does not take into account a very thorough assessment of all the evidence. P(condition | evidence) is not very accurate when the evidence is vague and sounds like many other conditions. It is a rare doctor who spends more than a few minutes asking about all possible symptoms, even those that may not occur to the patient that they have. Such doctors certainly exist, and I’ve had a few, but for the most part I have to do my own research and present it to a doctor for it to even be considered as a possibility. As anecdata of one, I’ve been having all sorts of weird rashes that come and go. Could be eczema, lupus, psoriasis, who knows. Googling makes you think it could be anything. So I took my entire medical history, all medications, all symptoms, their frequency, etc. and threw everything into ChatGPT with the prompt “Give me a list of possible diagnoses ranked by likelihood based on the evidence”. At the top of the list “Mononucleosis (EBV induced)”. Ah, but I forgot to mention that I had mono 12 years ago. I can’t catch that twice. So I told ChatGPT that I had it previously. “Still mono. Possibly recurrent due to some problem of the immune system.” So I went and got an EBV antibody panel. Positive for all three, including the EBNA test which is positive if you ever had mono previously. It blew my mind that a LLM sussed this out, because mono wasn’t even on my radar.

dartos 2y ago
It’s not just unpopular, it’s dangerous. Please people, for the love of life and health, don’t trust your medical issues to LLMs. You’re lucky it was just mono. Maybe find a better doctor if you feel you’re not getting good enough care…
- atonse 2y ago
  I agree with this. Any coder can tell you that AI will write some decent code, but it takes (at least at this point in time) an experienced coder to tell "this is crap code" or "this is decent code" – find a better doctor, because it's going to be hard to tell whether the advice it's giving you is the equivalent of crap code or good quality code.
- throwaway153371 2y ago
  I guess I don’t understand how it’s dangerous. I’m not giving myself any kind of independent treatment based on the output of an LLM. But it provides possibilities that doctors may have overlooked. Those doctors can then empirically assess the possibility in detail to determine whether it is correct.
  - jasonjmcghee 2y ago
    I think parent comment is suggesting it's dangerous for you to publicly tell others that ChatGPT is better than their doctor at diagnosing them (however little of your doctor's attention you get) regardless of whether there is or isn't truth to it in certain cases. People can't know when they should/can trust an llm over a doctor. People will also misuse LLMs in the context of medical diagnosis where asking the right questions and examining the right places (and properly) can be critical. People have biases, people can have psychosomatic symptoms, people can be easily influenced, etc.
    - dartos 2y ago
      > I think parent comment is suggesting it's dangerous for you to publicly tell others that ChatGPT is better than their doctor at diagnosing them (however little of your doctor's attention you get) regardless of whether there is or isn't truth to it in certain cases. Yes, exactly. Doubly so when one has no medical expertise.
  - bombcar 2y ago
    The danger is on the other side, you give the AI all your symptoms, it tells you you're fine, nothing is wrong, don't do anything. And then you ignore it and don't go to the doctor and fall over dead. Unlikely, probably. ChatGPT aims to please. There's also the risk that it recommends something that is in appropriate, but within your ability (like over-the-counter meds that shouldn't be used in your condition). But - all this stuff happens all the time anyway. Most people don't involve a doctor until they encounter something really wrong that they don't recognize. They're more likely to mention it to someone else offhand and may take whatever recommendation they get.
- causal 2y ago
  I think there are two ways using LLMs for medical advice are dangerous: 1) Treating oneself after getting bad advice or 2) Not getting treatment at all because an LLM thought a critical condition was normal. But the article and the comment both describe situations where they talked to a physician first, went to ChatGPT to refine their understanding, and then went back to a physician. Seems like a good use of LLMs.
  - dartos 2y ago
    OC started their comment with > Unpopular take: ChatGPT is better at diagnosing medical problems than a doctor who does a cursory evaluation. Which is dangerous and harmful advice. No reason to muddy the waters.
- deleted 2y ago
  [deleted]
- ebiester 2y ago
  It must be wonderful to live in a place where that's an option. Due to my partner's work, I live in a town of about 100,000. There is a doctor shortage - New patient intake can be months. The system hangs on with physicians' assistants. Going to a specialist often means months of waiting or hours of travel. We are living in a degraded health system between the lack of doctors, burnout of doctors and nurses, and a private equity system buying up everything in sight and sucking out every last dollar it can. Maybe it's better in large cities.
  - dartos 2y ago
    I don’t live in a large city. I’d bet that, if you live in the US, there are options for doctors within a 1-2 hour drive from you. I drive an hour and a half to see an independent dentist I trust. I’m lucky enough to have a decent doctor only 20 mins away. Either way, I’d never trust my health to an automated system. Especially one which can’t be held accountable in any way.
- tim333 2y ago
  Depends on how you use it. Just using LLMs is probably not good but LLM + seeing the doctor is quite likely better than just seeing a doctor occasionally. At least LLM advice is normally quite sensible compared with a lot of the junk advice out on the internet.
quantumwoke 2y ago
This doesn't seem relevant to the post, rather an anecdote where you confirmed that you had EBV in the past with the help(?) of ChatGPT (which you already knew). I don't see where you diagnosed the cause of your rashes.
- throwaway153371 2y ago
  I didn’t. I followed up with a doctor and mentioned the possibility of mono who then performed the EBV tests and provided the official diagnosis after previously seeing a different doctor who incorrectly diagnosed the problem as eczema.
famouswaffles 2y ago
What research there is about SOTA LLMs and diagnoses does suggest they are really good.

dartos 2y ago

It’s not just unpopular, it’s dangerous. Please people, for the love of life and health, don’t trust your medical issues to LLMs. You’re lucky it was just mono. Maybe find a better doctor if you feel you’re not getting good enough care…

atonse 2y ago
I agree with this. Any coder can tell you that AI will write some decent code, but it takes (at least at this point in time) an experienced coder to tell "this is crap code" or "this is decent code" – find a better doctor, because it's going to be hard to tell whether the advice it's giving you is the equivalent of crap code or good quality code.
throwaway153371 2y ago
I guess I don’t understand how it’s dangerous. I’m not giving myself any kind of independent treatment based on the output of an LLM. But it provides possibilities that doctors may have overlooked. Those doctors can then empirically assess the possibility in detail to determine whether it is correct.
- jasonjmcghee 2y ago
  I think parent comment is suggesting it's dangerous for you to publicly tell others that ChatGPT is better than their doctor at diagnosing them (however little of your doctor's attention you get) regardless of whether there is or isn't truth to it in certain cases. People can't know when they should/can trust an llm over a doctor. People will also misuse LLMs in the context of medical diagnosis where asking the right questions and examining the right places (and properly) can be critical. People have biases, people can have psychosomatic symptoms, people can be easily influenced, etc.
  - dartos 2y ago
    > I think parent comment is suggesting it's dangerous for you to publicly tell others that ChatGPT is better than their doctor at diagnosing them (however little of your doctor's attention you get) regardless of whether there is or isn't truth to it in certain cases. Yes, exactly. Doubly so when one has no medical expertise.
- bombcar 2y ago
  The danger is on the other side, you give the AI all your symptoms, it tells you you're fine, nothing is wrong, don't do anything. And then you ignore it and don't go to the doctor and fall over dead. Unlikely, probably. ChatGPT aims to please. There's also the risk that it recommends something that is in appropriate, but within your ability (like over-the-counter meds that shouldn't be used in your condition). But - all this stuff happens all the time anyway. Most people don't involve a doctor until they encounter something really wrong that they don't recognize. They're more likely to mention it to someone else offhand and may take whatever recommendation they get.
causal 2y ago
I think there are two ways using LLMs for medical advice are dangerous: 1) Treating oneself after getting bad advice or 2) Not getting treatment at all because an LLM thought a critical condition was normal. But the article and the comment both describe situations where they talked to a physician first, went to ChatGPT to refine their understanding, and then went back to a physician. Seems like a good use of LLMs.
- dartos 2y ago
  OC started their comment with > Unpopular take: ChatGPT is better at diagnosing medical problems than a doctor who does a cursory evaluation. Which is dangerous and harmful advice. No reason to muddy the waters.
deleted 2y ago
[deleted]
ebiester 2y ago
It must be wonderful to live in a place where that's an option. Due to my partner's work, I live in a town of about 100,000. There is a doctor shortage - New patient intake can be months. The system hangs on with physicians' assistants. Going to a specialist often means months of waiting or hours of travel. We are living in a degraded health system between the lack of doctors, burnout of doctors and nurses, and a private equity system buying up everything in sight and sucking out every last dollar it can. Maybe it's better in large cities.
- dartos 2y ago
  I don’t live in a large city. I’d bet that, if you live in the US, there are options for doctors within a 1-2 hour drive from you. I drive an hour and a half to see an independent dentist I trust. I’m lucky enough to have a decent doctor only 20 mins away. Either way, I’d never trust my health to an automated system. Especially one which can’t be held accountable in any way.
tim333 2y ago
Depends on how you use it. Just using LLMs is probably not good but LLM + seeing the doctor is quite likely better than just seeing a doctor occasionally. At least LLM advice is normally quite sensible compared with a lot of the junk advice out on the internet.

quantumwoke 2y ago

This doesn't seem relevant to the post, rather an anecdote where you confirmed that you had EBV in the past with the help(?) of ChatGPT (which you already knew). I don't see where you diagnosed the cause of your rashes.

throwaway153371 2y ago
I didn’t. I followed up with a doctor and mentioned the possibility of mono who then performed the EBV tests and provided the official diagnosis after previously seeing a different doctor who incorrectly diagnosed the problem as eczema.

famouswaffles 2y ago

What research there is about SOTA LLMs and diagnoses does suggest they are really good.