3 ms·

Claude thinks it's a tumor, radiologists disagree

19 points by baxtr 3y ago

bthrn 3y ago
For some reason I read this and then think of Arnold saying, “it’s not a toomah!”
infecto 3y ago
1) We already know LLMs when given the opportunity, will often make up information. 2) I am not sure its surprising that a general purpose model is not great at reading MRI scans.
- striking 3y ago
  It's not surprising to you or me, but think of someone less technologically proficient. They won't know the difference, and if the AI complies with their request in a convincing way, they may very well believe it.
DeepYogurt 3y ago
Really contextualizes those stories that read as "we don't know how the AI found the medical issue, but it did!".
prashp 3y ago
Claude isn't an approved medical device so why is willing to provide a diagnosis?
- minimaxir 3y ago
  Because it isn't RLHFed to do so. It's a good example why alignment sometimes isn't a bad thing. ChatGPT does specifically disallow medical advice and generally it is RLHFed not to provide it: https://openai.com/policies/usage-policies https://openai.com/policies/usage-policies
- SkyPuncher 3y ago
  My friends aren’t licensed healthcare providers, but they’re more than willing to provide diagnoses.
- rsynnott 3y ago
  What does ‘willing’ mean here? To be clear, these things are not people, or anything resembling people. They do not have will.
GaggiX 3y ago
To be honest Claude vision is not that great, it's probably Claude’s biggest weakness compare to GPT-4.
deleted 3y ago
[deleted]
olliej 3y ago
Yeah, there was a paper about AI excellence in diagnosing radiology images a few (5-10 now?) years ago, that turned out to be largely driven by accuracy skewed bias in the input data (IIRC it was using the existence of lab-style lighting, rulers, etc as its diagnosis). Chat-GPT and all the LLMs are doing essentially the same "predict the next work in a sentence" to sound as plausibly real as possible, which is essentially the same gimmick, only we call it hallucination due to VC funded AI firms need to pretend LLMs understand anything.
Krasnol 3y ago
There are much better AI approaches to the problem like: https://mediaire.ai/en/mdbrain/ https://mediaire.ai/en/mdbrain/
resource0x 3y ago
"We should stop training radiologists now. It's just completely obvious that within five years, deep learning is going to do better than radiologists" G.Hinton, 2016
- viraptor 3y ago
  https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6483404/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6483404/ > The error rate produced by a deep learning based methods dropped below that achieved by human observers in 2015 for the first time Essentially we just need good implementations at this point. We're there, just not widely deployed.
rvz 3y ago
On some days, Claude (and other LLMs) will hallucinate on one input and produce garbage and in the next update it will then believe something else in the next unpredictably. Perhaps using general purpose LLMs for safety critical use cases without humans isn't a good idea. This is why we have a 'human in the loop' giving the feedback to resolve the trust issue that LLMs lack. Maybe we should trust LLMs like Claude to pilot a plane without any human pilots on board and raise VC money on the idea on replacing all pilots since we keep having these conversations on LLMs being compared with humans and countless VCs screaming over robots replacing humans in all occupations and 'accelerating everything' which could not be more wrong.
rsynnott 3y ago
Bloody hell, this sort of thing is going to get people killed. The magic box is not a person, let alone a qualified doctor. Do not ask the magic box questions about anything important.
cheema33 3y ago
My girlfriend has had some medical issues for a few years and none of the 3-4 doctors she has seen have been able to diagnose her. That is not to say that no doctor can. Just not the ones she has seen. I do not believe in the infallibility of medical professionals. And I have seen AI totally get it wrong every day. But, I am of the opinion that we are not very far from the point where AI will be better at diagnosing issues than an average doctor. That isn't a high bar. An average doctor isn't very good. I think AI is almost there already for an average programmer. An average programmer isn't very good either.

19 points by baxtr 3y ago

For some reason I read this and then think of Arnold saying, “it’s not a toomah!”

1) We already know LLMs when given the opportunity, will often make up information. 2) I am not sure its surprising that a general purpose model is not great at reading MRI scans.

striking 3y ago
It's not surprising to you or me, but think of someone less technologically proficient. They won't know the difference, and if the AI complies with their request in a convincing way, they may very well believe it.

DeepYogurt 3y ago

Really contextualizes those stories that read as "we don't know how the AI found the medical issue, but it did!".

prashp 3y ago

Claude isn't an approved medical device so why is willing to provide a diagnosis?

minimaxir 3y ago
Because it isn't RLHFed to do so. It's a good example why alignment sometimes isn't a bad thing. ChatGPT does specifically disallow medical advice and generally it is RLHFed not to provide it: https://openai.com/policies/usage-policies https://openai.com/policies/usage-policies
SkyPuncher 3y ago
My friends aren’t licensed healthcare providers, but they’re more than willing to provide diagnoses.
rsynnott 3y ago
What does ‘willing’ mean here? To be clear, these things are not people, or anything resembling people. They do not have will.

GaggiX 3y ago

To be honest Claude vision is not that great, it's probably Claude’s biggest weakness compare to GPT-4.

deleted 3y ago

[deleted]

olliej 3y ago

Yeah, there was a paper about AI excellence in diagnosing radiology images a few (5-10 now?) years ago, that turned out to be largely driven by accuracy skewed bias in the input data (IIRC it was using the existence of lab-style lighting, rulers, etc as its diagnosis). Chat-GPT and all the LLMs are doing essentially the same "predict the next work in a sentence" to sound as plausibly real as possible, which is essentially the same gimmick, only we call it hallucination due to VC funded AI firms need to pretend LLMs understand anything.

Krasnol 3y ago

There are much better AI approaches to the problem like: https://mediaire.ai/en/mdbrain/ https://mediaire.ai/en/mdbrain/

resource0x 3y ago

"We should stop training radiologists now. It's just completely obvious that within five years, deep learning is going to do better than radiologists" G.Hinton, 2016

viraptor 3y ago
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6483404/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6483404/ > The error rate produced by a deep learning based methods dropped below that achieved by human observers in 2015 for the first time Essentially we just need good implementations at this point. We're there, just not widely deployed.

rvz 3y ago

On some days, Claude (and other LLMs) will hallucinate on one input and produce garbage and in the next update it will then believe something else in the next unpredictably. Perhaps using general purpose LLMs for safety critical use cases without humans isn't a good idea. This is why we have a 'human in the loop' giving the feedback to resolve the trust issue that LLMs lack. Maybe we should trust LLMs like Claude to pilot a plane without any human pilots on board and raise VC money on the idea on replacing all pilots since we keep having these conversations on LLMs being compared with humans and countless VCs screaming over robots replacing humans in all occupations and 'accelerating everything' which could not be more wrong.

rsynnott 3y ago

Bloody hell, this sort of thing is going to get people killed. The magic box is not a person, let alone a qualified doctor. Do not ask the magic box questions about anything important.

cheema33 3y ago

My girlfriend has had some medical issues for a few years and none of the 3-4 doctors she has seen have been able to diagnose her. That is not to say that no doctor can. Just not the ones she has seen. I do not believe in the infallibility of medical professionals. And I have seen AI totally get it wrong every day. But, I am of the opinion that we are not very far from the point where AI will be better at diagnosing issues than an average doctor. That isn't a high bar. An average doctor isn't very good. I think AI is almost there already for an average programmer. An average programmer isn't very good either.