3 ms·

Nightmare because they're always right and the A.I second guessing is always wrong, or because they just don't like to be second guessed?

vimda 3mo ago
Nightmare because users approach LLMs with the false confidence that they're always right, and present LLM outputs as fact to Doctors who have to waste time explaining that it's wrong most of the time. It hurts more than it helps.
- weatherlite 3mo ago
  > It hurts more than it helps Hurts who ? Yes the doctor is super stressed and has maybe 10 minutes for you that's the actual problem, it's not like before LLMs they were super glad to sit there and answer all your questions.
drw85 3mo ago
Nightmare because the AI is just generating a random text that fits the question.
- betaby 3mo ago
  I feel the same when visiting a doctor in Canada. In that 2 minutes I have with they in one appointment per year I hear a standard text.
- poszlem 3mo ago
  This is a very peculiar use of the word "random".
- Legend2440 3mo ago
  This is not a fair assessment of what AI is doing. Studies have found that newer reasoning AIs are about as good at diagnosing illness from a written description of symptoms as doctors are. Granted, it cannot actually examine a patient, so we're not replacing doctors anytime soon. But your view is obsolete. https://www.science.org/doi/10.1126/science.adz4433 https://www.science.org/doi/10.1126/science.adz4433
  - Retric 3mo ago
    They are using the “gold standard for the evaluation of expert medical computing systems” not a proxy for what a doctor actually does when diagnosing someone. It may have some utility after diagnosis, but this test doesn’t demonstrate utility for patients.
  - snackerblues 3mo ago
    [flagged]
    - microgpt 3mo ago
      But I, SCP-426, am a toaster.
- d1sxeyes 3mo ago
  Not quite. An LLM generates text that would likely follow. The sky is… “blue”. A patient in pain with a bone protruding from their shin has a… “broken leg”. The more training data, the more questions it can answer with a reasonable degree of probability of accuracy. Throwing away a potentially useful analysis just because it’s probabilistic seems a bit like throwing the baby out with the bath water.
  - drw85 3mo ago
    But for obvious cases like this, you don't need a second or first opinion. This case is about handing a 3D imaging result to a text predictor and hoping for a valid second opinion.
    - d1sxeyes 3mo ago
      Yes that’s my point. An LLM can clearly accurately predict obvious cases, so it’s reasonable to assume that it can predict less obvious cases with somewhat less accuracy. The real question is where’s the cut-off point between accuracy and utility. Remember: a second human opinion can also be wrong, and even a wrong opinion can still be useful (especially in medicine where differential diagnoses are a common practice - if the LLM gives you a useless opinion, you rule it out and move on). I don’t think it’s particularly unreasonable to think that an LLM would have enough literature, or enough reasoning ability, to be able to generate a plausible interpretation of the data. A human can then review and say either “yeah that’s clearly not the case here” or “hmm, actually that could explain it, maybe we should order another test”.
mixologic 3mo ago
Its a nightmare because it erodes trust. Doctors are not "always right" which is why "always get a second opinion" is codified in culture. But AI's problem is that its completely full of shit, sometimes, and the people most qualified to evaluate whether its full of shit are the doctors, not the patients, but just like OP's original article, patients are left feeling like their second opinion from AI might be more trustworthy than their doctors opinion.
- simianwords 3mo ago
  The notion that only doctors can verify is false! Doctors are better at verification but normal people can also verify. This is just empirically true. Examples of things normal people can verify - procedural errors that Claude can capture like some blatantly high dosage (grams instead of milligrams) - outdated treatment plan, maybe there’s a credible new treatment plan that’s been used for years but the doctors were not updated - literally being injected homeopathic drugs (takes no smart person to flag this) Let’s stop talking as if doctors have a divine right here. And let’s accept some agency.
  - drw85 3mo ago
    But those are obvious errors. What if the AI tells you to up your intake of X and it seems plausible? So you up your intake of X by taking some supplement, but upping the intake of X makes your body deplete more of Y and now you have a new or compounding problem. A doctor might have never recommended upping X, because they would know what it does to your body. Or they might have suggested additional supplementation to avoid this. The fact that LLMs are trained on all public knowledge is a huge red flag, because there are more wrong infos out there than right ones. Especially about health, diet, etc.
    - simianwords 3mo ago
      I was about to respond but your last statement implies fundamental misunderstanding of how LLMs work. I don’t think you even know about RLHF and you think good and bad ideas are spread in proportion to how much they are seen on the internet.
tuvix 3mo ago
There’s more than two options here. It was already difficult to deal with self diagnosis for doctors, now we have a machine that outputs recommendations, and does it with confidence whether it’s correct or not. The same issues that were present with search-engine self diagnosis are still present with LLMs. If you provide Google with an incomplete list of symptoms and can’t interpret the information you find correctly, you will likely get an incorrect diagnosis. The same is true for LLM output.
- rvnx 3mo ago
  There are quite a few disclaimers everywhere that soften confidence: "always ask a medical specialist", "I'm not a doctor", "this could have been this or that but really not sure", etc.
  - neonstatic 3mo ago
    No one cares about this, especially those who believe the machine. It's just there for the provider to avoid responsibility.
- dheera 3mo ago
  Everyone on the internet loves to put doctors on a pedestal, but I think upwards of 30% of my doctor visits have been misdiagnosed. There's a reason I ask AI about absolutely everything medical and there's a reason I keep extra quantities of prescription medications around for emergencies. I've saved my own ass a lot more times than the doctors have, thanks to good doctors not being available.
  - ricardobayes 3mo ago
    That works in your case because you likely already have an analytical mindset, can reliably discard or research information further. Same thing why AI is in an enabler for people having already some skill. But it can be very dangerous or misleading for people blindly believing all output.
- weatherlite 3mo ago
b800h 3mo ago
Well it was a nightmare for my mother's do-nothing GP surgery in the UK. She had several conditions which were being handled completely separately without central coordination, and her health was in serious decline. We went in with a list of 20 AI-generated questions based on her conditions and treatment (which I was able to screen as I have a bio postgrad, but not medical training), including those related to NICE guidelines and procedure, and, frankly the GP bricked it and ordered a load of new interventions. My mother started to get proper treatment. I wouldn't trust AI to make a diagnosis, but I would absolutely trust it to notice where procedure hasn't been correctly followed, where a treatment is counter-indicated because someone has missed a line on a health record, or where there's a clear potential alternate diagnosis which has been missed for spurious reasons. Also, unfortunately, where doctors aren't doing a decent job - often because they're overworked or underfunded.
- ricardobayes 3mo ago
  UK has probably the worst healthcare in the developed world. In part perhaps due to UK blindly accepting any kind of medical degree (doctors and nurses) from all over the globe. Yes, you heard that right, they verify the validity of the degree but there is no formal standardized exam to sit to practice in the UK.
  - b800h 3mo ago
    Her nurses frequently couldn't speak English. I had to spell out her medical conditions letter by letter so that the nurse could input it onto the medical system. She was entering gibberish / wrong info until I stepped in, as I was watching the PAS out of interest.
    - weatherlite 3mo ago
      fucking hell. dont they have language tests for foreign nurses? how the hell did she pass it ?
      - b800h 3mo ago

vimda 3mo ago

Nightmare because users approach LLMs with the false confidence that they're always right, and present LLM outputs as fact to Doctors who have to waste time explaining that it's wrong most of the time. It hurts more than it helps.

weatherlite 3mo ago
> It hurts more than it helps Hurts who ? Yes the doctor is super stressed and has maybe 10 minutes for you that's the actual problem, it's not like before LLMs they were super glad to sit there and answer all your questions.

drw85 3mo ago

Nightmare because the AI is just generating a random text that fits the question.

betaby 3mo ago
I feel the same when visiting a doctor in Canada. In that 2 minutes I have with they in one appointment per year I hear a standard text.
poszlem 3mo ago
This is a very peculiar use of the word "random".
Legend2440 3mo ago
This is not a fair assessment of what AI is doing. Studies have found that newer reasoning AIs are about as good at diagnosing illness from a written description of symptoms as doctors are. Granted, it cannot actually examine a patient, so we're not replacing doctors anytime soon. But your view is obsolete. https://www.science.org/doi/10.1126/science.adz4433 https://www.science.org/doi/10.1126/science.adz4433
- Retric 3mo ago
  They are using the “gold standard for the evaluation of expert medical computing systems” not a proxy for what a doctor actually does when diagnosing someone. It may have some utility after diagnosis, but this test doesn’t demonstrate utility for patients.
- snackerblues 3mo ago
  [flagged]
  - microgpt 3mo ago
    But I, SCP-426, am a toaster.
d1sxeyes 3mo ago
Not quite. An LLM generates text that would likely follow. The sky is… “blue”. A patient in pain with a bone protruding from their shin has a… “broken leg”. The more training data, the more questions it can answer with a reasonable degree of probability of accuracy. Throwing away a potentially useful analysis just because it’s probabilistic seems a bit like throwing the baby out with the bath water.
- drw85 3mo ago
  But for obvious cases like this, you don't need a second or first opinion. This case is about handing a 3D imaging result to a text predictor and hoping for a valid second opinion.
  - d1sxeyes 3mo ago
    Yes that’s my point. An LLM can clearly accurately predict obvious cases, so it’s reasonable to assume that it can predict less obvious cases with somewhat less accuracy. The real question is where’s the cut-off point between accuracy and utility. Remember: a second human opinion can also be wrong, and even a wrong opinion can still be useful (especially in medicine where differential diagnoses are a common practice - if the LLM gives you a useless opinion, you rule it out and move on). I don’t think it’s particularly unreasonable to think that an LLM would have enough literature, or enough reasoning ability, to be able to generate a plausible interpretation of the data. A human can then review and say either “yeah that’s clearly not the case here” or “hmm, actually that could explain it, maybe we should order another test”.

mixologic 3mo ago

Its a nightmare because it erodes trust. Doctors are not "always right" which is why "always get a second opinion" is codified in culture. But AI's problem is that its completely full of shit, sometimes, and the people most qualified to evaluate whether its full of shit are the doctors, not the patients, but just like OP's original article, patients are left feeling like their second opinion from AI might be more trustworthy than their doctors opinion.

simianwords 3mo ago
The notion that only doctors can verify is false! Doctors are better at verification but normal people can also verify. This is just empirically true. Examples of things normal people can verify - procedural errors that Claude can capture like some blatantly high dosage (grams instead of milligrams) - outdated treatment plan, maybe there’s a credible new treatment plan that’s been used for years but the doctors were not updated - literally being injected homeopathic drugs (takes no smart person to flag this) Let’s stop talking as if doctors have a divine right here. And let’s accept some agency.
- drw85 3mo ago
  But those are obvious errors. What if the AI tells you to up your intake of X and it seems plausible? So you up your intake of X by taking some supplement, but upping the intake of X makes your body deplete more of Y and now you have a new or compounding problem. A doctor might have never recommended upping X, because they would know what it does to your body. Or they might have suggested additional supplementation to avoid this. The fact that LLMs are trained on all public knowledge is a huge red flag, because there are more wrong infos out there than right ones. Especially about health, diet, etc.
  - simianwords 3mo ago
    I was about to respond but your last statement implies fundamental misunderstanding of how LLMs work. I don’t think you even know about RLHF and you think good and bad ideas are spread in proportion to how much they are seen on the internet.

tuvix 3mo ago

There’s more than two options here. It was already difficult to deal with self diagnosis for doctors, now we have a machine that outputs recommendations, and does it with confidence whether it’s correct or not. The same issues that were present with search-engine self diagnosis are still present with LLMs. If you provide Google with an incomplete list of symptoms and can’t interpret the information you find correctly, you will likely get an incorrect diagnosis. The same is true for LLM output.

rvnx 3mo ago
There are quite a few disclaimers everywhere that soften confidence: "always ask a medical specialist", "I'm not a doctor", "this could have been this or that but really not sure", etc.
- neonstatic 3mo ago
  No one cares about this, especially those who believe the machine. It's just there for the provider to avoid responsibility.
dheera 3mo ago
Everyone on the internet loves to put doctors on a pedestal, but I think upwards of 30% of my doctor visits have been misdiagnosed. There's a reason I ask AI about absolutely everything medical and there's a reason I keep extra quantities of prescription medications around for emergencies. I've saved my own ass a lot more times than the doctors have, thanks to good doctors not being available.
- ricardobayes 3mo ago
  That works in your case because you likely already have an analytical mindset, can reliably discard or research information further. Same thing why AI is in an enabler for people having already some skill. But it can be very dangerous or misleading for people blindly believing all output.
weatherlite 3mo ago

b800h 3mo ago

Well it was a nightmare for my mother's do-nothing GP surgery in the UK. She had several conditions which were being handled completely separately without central coordination, and her health was in serious decline. We went in with a list of 20 AI-generated questions based on her conditions and treatment (which I was able to screen as I have a bio postgrad, but not medical training), including those related to NICE guidelines and procedure, and, frankly the GP bricked it and ordered a load of new interventions. My mother started to get proper treatment. I wouldn't trust AI to make a diagnosis, but I would absolutely trust it to notice where procedure hasn't been correctly followed, where a treatment is counter-indicated because someone has missed a line on a health record, or where there's a clear potential alternate diagnosis which has been missed for spurious reasons. Also, unfortunately, where doctors aren't doing a decent job - often because they're overworked or underfunded.

ricardobayes 3mo ago
UK has probably the worst healthcare in the developed world. In part perhaps due to UK blindly accepting any kind of medical degree (doctors and nurses) from all over the globe. Yes, you heard that right, they verify the validity of the degree but there is no formal standardized exam to sit to practice in the UK.
- b800h 3mo ago
  Her nurses frequently couldn't speak English. I had to spell out her medical conditions letter by letter so that the nurse could input it onto the medical system. She was entering gibberish / wrong info until I stepped in, as I was watching the PAS out of interest.
  - weatherlite 3mo ago
    fucking hell. dont they have language tests for foreign nurses? how the hell did she pass it ?
    - b800h 3mo ago