5 ms·

I don't want to single you out, but what should I be taking away from the often recited "well people sometimes* can't do thing X either"-counterargument? Is all

by rzzzt 3y ago

I don't want to single you out, but what should I be taking away from the often recited "well people sometimes* can't do thing X either"-counterargument? Is all of this just fine? Can't we expect just a little bit more, I don't know, accuracy or rigor from a computer than a living person?

chasd00 3y ago
Yeah I don’t buy that argument either. If excel got the math wrong replying with “well humans do to” wouldn’t fly.
atleastoptimal 3y ago
We've already passed the point where LLM's are better than human experts for medical diagnoses. In fact, according to this study, even LLM's alone are more accurate than human experts + LLM's, meaning any input the humans added was only a detriment to the accuracy https://arxiv.org/pdf/2312.00164.pdf https://arxiv.org/pdf/2312.00164.pdf (Page 8) Computers are already perfectly accurate, and have been for decades in explicit quantifiable fields. In medicine, since a computer cannot perfectly replicate every single cell in the human body, its abstractions will be lower resolution than reality, but what matters is whether that low resolution abstraction is better than the alternative (human doctors). A human doctor couldn't bring up a list of citations in literature instantly regarding a diagnosis. A LLM can.
- rzzzt 3y ago
  They could, but they don't, at least that is what I'm getting from the article. Even if they do, someone with the capability and understanding required (ie. not me) needs to bring that source up and verify that the claims align with the citation; the authors decided to use GPT-4 for this: "We adapted GPT-4 to verify whether sources substantiate statements and found the approach to be surprisingly reliable." I'm not happy with that either.
- deleted 3y ago
  [deleted]
- staticman2 3y ago
  Even if that paper hadn't said >>>"We are therefore very cautious to extrapolate our findings toward any implications about the LLM’s utility as a standalone diagnostic tool" Your post would be a extraordinary claim and need extraordinary evidence, not a specific study of a specific scenario.
  - atleastoptimal 3y ago
    there already is extraordinary evidence of GPT-4's superhuman diagnostic abilities. One anecdotal example is this: https://www.today.com/health/mom-chatgpt-diagnosis-pain-rcna101843 https://www.today.com/health/mom-chatgpt-diagnosis-pain-rcna... Another study regarding GPT-4 beating human experts: https://www.medrxiv.org/content/10.1101/2023.04.20.23288859v2.full.pdf https://www.medrxiv.org/content/10.1101/2023.04.20.23288859v... Lots of data is pointing to the same conclusion: GPT-4 is at least as good if not better than human experts in medical diagnosis, at least in the areas studied. Thus the probability of a correct diagnosis is higher, thus safer, with GPT-4 than with any individual human expert.
    - staticman2 3y ago
      This is so silly, the one study you linked to says that GPT 4 may have been trained on the answers to the test they gave it. So smart. And since GPT 4 can't examine a patient's body the claim it's better at diagnosis that a human doctor seems like such a wacky thing to search the internet for "studies" to prove in the first place.
      - atleastoptimal 3y ago
        A nurse can examine a patient's body. Medical tools can and report their diagnostics with high precision. GPT-4 is multi-modal. I feel you are nitpicking because you don't like the idea of an LLM being better than a human expert. Even if they weren't better than doctors nowadays, the chance they won't be in 1-2 years is tiny.
        staticman2 3y ago
- TacticalCoder 3y ago
  > A human doctor couldn't bring up a list of citations in literature instantly regarding a diagnosis. A LLM can. TFA is, literally, about LLMs spouting out erroneous medical references. I don't care about made up medical references or court cases. I'm sure there are ways to bring up instantly a list of publication regarding a diagnosis (which a LLM may or may in the future correctly give: the diagnosis I mean) but I'm really not sure a LLM is what's needed to do then generate the list of related publication. I mean, FFS, they are compressed, lossy, knowledge. LLMs are going to become tools as part of a toolchain. They're not a panacea.
- __loam 3y ago
  > A human doctor couldn't bring up a list of citations in literature instantly regarding a diagnosis. A LLM can. Which it will happily make up.
- xetplan 3y ago
  People have a delusional, religious type faith in their own doctor. It is perfectly understandable given the context. The average person has no idea what iatrogenesis is. Even if you try to explain this concept, the average person doesn't want to understand the idea. The estimates are 10-15% of people admitted to a hospital will suffer iatrogenic harm.

by rzzzt 3y ago

chasd00 3y ago

Yeah I don’t buy that argument either. If excel got the math wrong replying with “well humans do to” wouldn’t fly.

atleastoptimal 3y ago

We've already passed the point where LLM's are better than human experts for medical diagnoses. In fact, according to this study, even LLM's alone are more accurate than human experts + LLM's, meaning any input the humans added was only a detriment to the accuracy https://arxiv.org/pdf/2312.00164.pdf https://arxiv.org/pdf/2312.00164.pdf (Page 8) Computers are already perfectly accurate, and have been for decades in explicit quantifiable fields. In medicine, since a computer cannot perfectly replicate every single cell in the human body, its abstractions will be lower resolution than reality, but what matters is whether that low resolution abstraction is better than the alternative (human doctors). A human doctor couldn't bring up a list of citations in literature instantly regarding a diagnosis. A LLM can.

rzzzt 3y ago
They could, but they don't, at least that is what I'm getting from the article. Even if they do, someone with the capability and understanding required (ie. not me) needs to bring that source up and verify that the claims align with the citation; the authors decided to use GPT-4 for this: "We adapted GPT-4 to verify whether sources substantiate statements and found the approach to be surprisingly reliable." I'm not happy with that either.
deleted 3y ago
[deleted]
staticman2 3y ago
Even if that paper hadn't said >>>"We are therefore very cautious to extrapolate our findings toward any implications about the LLM’s utility as a standalone diagnostic tool" Your post would be a extraordinary claim and need extraordinary evidence, not a specific study of a specific scenario.
- atleastoptimal 3y ago
  there already is extraordinary evidence of GPT-4's superhuman diagnostic abilities. One anecdotal example is this: https://www.today.com/health/mom-chatgpt-diagnosis-pain-rcna101843 https://www.today.com/health/mom-chatgpt-diagnosis-pain-rcna... Another study regarding GPT-4 beating human experts: https://www.medrxiv.org/content/10.1101/2023.04.20.23288859v2.full.pdf https://www.medrxiv.org/content/10.1101/2023.04.20.23288859v... Lots of data is pointing to the same conclusion: GPT-4 is at least as good if not better than human experts in medical diagnosis, at least in the areas studied. Thus the probability of a correct diagnosis is higher, thus safer, with GPT-4 than with any individual human expert.
  - staticman2 3y ago
    This is so silly, the one study you linked to says that GPT 4 may have been trained on the answers to the test they gave it. So smart. And since GPT 4 can't examine a patient's body the claim it's better at diagnosis that a human doctor seems like such a wacky thing to search the internet for "studies" to prove in the first place.
    - atleastoptimal 3y ago
      A nurse can examine a patient's body. Medical tools can and report their diagnostics with high precision. GPT-4 is multi-modal. I feel you are nitpicking because you don't like the idea of an LLM being better than a human expert. Even if they weren't better than doctors nowadays, the chance they won't be in 1-2 years is tiny.
      - staticman2 3y ago
TacticalCoder 3y ago
> A human doctor couldn't bring up a list of citations in literature instantly regarding a diagnosis. A LLM can. TFA is, literally, about LLMs spouting out erroneous medical references. I don't care about made up medical references or court cases. I'm sure there are ways to bring up instantly a list of publication regarding a diagnosis (which a LLM may or may in the future correctly give: the diagnosis I mean) but I'm really not sure a LLM is what's needed to do then generate the list of related publication. I mean, FFS, they are compressed, lossy, knowledge. LLMs are going to become tools as part of a toolchain. They're not a panacea.
__loam 3y ago
> A human doctor couldn't bring up a list of citations in literature instantly regarding a diagnosis. A LLM can. Which it will happily make up.
xetplan 3y ago
People have a delusional, religious type faith in their own doctor. It is perfectly understandable given the context. The average person has no idea what iatrogenesis is. Even if you try to explain this concept, the average person doesn't want to understand the idea. The estimates are 10-15% of people admitted to a hospital will suffer iatrogenic harm.