5 ms·

This is the only part of AI that actually terrifies me. I’ve run into people on this very site who use LLMs as a doctor, asking it medical questions and follow

by 2023throwawayy 3y ago

This is the only part of AI that actually terrifies me. I’ve run into people on this very site who use LLMs as a doctor, asking it medical questions and following its advice. The same LLMs that hallucinate court cases when asked about law. The same LLMs that can’t perform basic arithmetic in a reliable fashion. The same LLMs that can’t process internally consistent logic. People are following the medical “advice” that comes out of these things. It will lead to deaths, no questions asked.

rolisz 3y ago
Following the advice of chatgpt without double checking? Bad idea. Using ChatGPT as a starting point? Sounds really good to me, been there, done that.
- twayt 3y ago
  Yea I think this is the most reasonable take. You can always check information before believing or acting on it. However it’s often super difficult to even get started and know what it is that you should be reading more about.
leetharris 3y ago
The reality is that the majority of things people want to go to the doctor for are not serious. If this can help with that, I am all for it.
bilsbie 3y ago
On the contrary modern medicine terrifies me. Something like this might be our only hope.
- geek_at 3y ago
  Chat GPT and that Amazon Healthcare thing will be more efficient than the US Healthcare system. Which is kind of crazy
- firebot 3y ago
  It should. Most medicine is just extracting plant chemicals, modifying them, concentrating them, and thereby they can patent what nature has provided.
bilsbie 3y ago
Wait until you hear about search engines …
techwizrd 3y ago
I used to work on a healthcare AI chatbot startup before traditional LLMs like BERT. We were definitely worried about accuracy and reliability of the medical advice then, and we had clinicians working closely to make sure the dialog trees were trustworthy. I work in aerospace medicine and aviation safety now, and I constantly encounter inadvisable use of LLMs and a lack of effective evaluation methods (especially for domain-specific LLMs). I appreciate the advisory notice in the README and the recommendation against using this in settings that may impact people. I sincerely hope that it's used ethically and responsibly.
ryandvm 3y ago
Sure, but we already have 250,000 medical deaths PER YEAR in the US due to medical errors (https://pubmed.ncbi.nlm.nih.gov/28186008/ https://pubmed.ncbi.nlm.nih.gov/28186008/). I don't think people should trust LLMs completely, but let's be real, they shouldn't trust humans completely either.
- blipmusic 3y ago
  Isn’t that whataboutism at its best? Those two things are completely unrelated.
  - mannyv 3y ago
    No, it's showing that the risk of errors exists even without AI. AI doesn't necessarily make that risk higher or lower a priori. Plus if you knew how much of current medical practice exists without evidence you wouldn't be worrying about AI.
    - blipmusic 3y ago
      Maybe it’s ok to worry about both? Not trusting ”arbitrary thing A” does not logically make ”arbitrary thing B” more trustworthy. I do realise that these models intend to (incrementally) represent collective knowledge and may get there in the future. But if you worry about A, why not worry about B which is based on A?
    - not2b 3y ago
      You seem to be assuming, without any evidence at all, that LLMs giving medical advice are likely to be roughly equivalent in accuracy to doctors who are actually examining the patient and not just processing language, just because you are aware that medical mistakes are common.
      - famouswaffles 3y ago
        https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10425828/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10425828/ Use of GPT-4 to Analyze Medical Records of Patients With Extensive Investigations and Delayed Diagnosis "Six patients 65 years or older (2 women and 4 men) were included in the analysis. The accuracy of the primary diagnoses made by GPT-4, clinicians, and Isabel DDx Companion was 4 of 6 patients (66.7%), 2 of 6 patients (33.3%), and 0 patients, respectively. If including differential diagnoses, the accuracy was 5 of 6 (83.3%) for GPT-4, 3 of 6 (50.0%) for clinicians, and 2 of 6 (33.3%) for Isabel DDx Companion"
davidjade 3y ago
Here’s a recent (yesterday) example of a benefit though. I tried unsuccessfully to search for an ECG analysis term (EAR or EA Run) using Google, DDG, etc. There was no magic set of quoting, search terms, etc. that could explain what those terms were. Ear is just too common for a word. ChatGPT however was able to take the context of the question I had (an ECG analysis) and lead me to the answer right away of what EAR meant. I wasn’t seeking medical advice though, just a better search engine with context. So there are clearly benefits here too.
- nhinck2 3y ago
  Ectopic Atrial Rhythm?
  - TaylorAlexander 3y ago
    Yeah I took the person's comment, snipped "ECG analysis term EAR meaning" from it and popped that in to google, found a page "ECG Glossary, Terms, & Terminology List" and under the E section it has "Ectopic Atrial Rhythm". That said if its correct, LLMs are less work for this kind of thing. But often when people explain to another person what they are looking for, as done in their comment, they will do a better job explaining what they need than when they are in their own head trying to google it. Which is why I just snipped their words to search for it.
deleted 3y ago
[deleted]
BrandoElFollito 3y ago
On the other hand, your MD is going to look for the obvious, or statistically relevant, or currently prominent disease. But they could be presented 99% probability for flu, 1% or wazalla, and that testing for wazalla means pinching your ear tout may actually be correctly diagnosed sometimes. It is not that MDs are incompetent, it is just that when wazalla was briefly mentioned during their studies, they happened to be in the toilets and missed it. Flu was mentioned 76 times because it is common. Disclaimer: I know medicine from "House, MD" but also witnessed a miraculous diagnosis on my father just because his MD happened to read an obscure article (for the story, he was diagnosed with a worm-induced illness that happened one or twice a year in France in the 80's. The worm was from a beach in Brazil, and my dad never travelled to Americas. He was kindly asked to provide a sample of blood to help research in France, which he did. Finally the drug to heal him was available in one pharmacy in Paris and in Lyon. We expected a hefty cost (though it is all covered in France), it costed 5 franks or so. But we were told with my brother to keep an eye on him as he may become delusional and try to jump through the window. The poor man cold hardly blink before we were on him:) Ah, and the pills were 2cm wide, looked like they were for an elephant. And he had 5 or so to swallow)
firebot 3y ago
What's to be terrified about? Humans also hallucinate. Doctors are terrible at their jobs.
- meroes 3y ago
  If doctors are terrible at their jobs so are programmers. And we should be terrified just as well at their AI creations then.
  - firebot 3y ago
    That's a fair point. Most programmers are terrible. As a lead programmer I've dealt with countless "programmers" that can not do their job without me spoon-feeding them code.
infecto 3y ago
I am personally excited for the possibilities. Nobody should be using a LLM without verifying. Will some people do it? Of course, I remember that court case where the lawyer used ChatGPT and it made up cases. If someone is going to make that mistake, there were other mistakes happening, not just using a LLM. On the positive note. LLMs offer the chance to potentially do much better diagnosing on hard to figure out cases.
renewiltord 3y ago
It's true. Only people like me should be allowed access to LLMs. Folks like you should be protected. Equivalent to accredited investor, there should be a tier of "knowledgeable normal person" who is allowed to do whatever.
brianjking 3y ago
In fairness, many doctors have terrible advice and make mistakes often. This is why malpractice insurance exists.
Exoristos 3y ago
Most deaths are already caused by the medical establishment.
ekianjo 3y ago
You heard about the bell curve concept? Chances are about half the doctors you see are at the lower part of the curve. Which means they are borderline or completely incompetent at what they do. I'd take my chances with a "properly trained" AI any day. Problem is, most medical corpus is full of bogus studies that have never been replicated, so it might be close to junk at this stage. > It will lead to deaths, regular doctors kill people everyday and get away with it because you accept the risks. What's different?

rolisz 3y ago

Following the advice of chatgpt without double checking? Bad idea. Using ChatGPT as a starting point? Sounds really good to me, been there, done that.

twayt 3y ago
Yea I think this is the most reasonable take. You can always check information before believing or acting on it. However it’s often super difficult to even get started and know what it is that you should be reading more about.

leetharris 3y ago

The reality is that the majority of things people want to go to the doctor for are not serious. If this can help with that, I am all for it.

bilsbie 3y ago

On the contrary modern medicine terrifies me. Something like this might be our only hope.

geek_at 3y ago
Chat GPT and that Amazon Healthcare thing will be more efficient than the US Healthcare system. Which is kind of crazy
firebot 3y ago
It should. Most medicine is just extracting plant chemicals, modifying them, concentrating them, and thereby they can patent what nature has provided.

bilsbie 3y ago

Wait until you hear about search engines …

techwizrd 3y ago

I used to work on a healthcare AI chatbot startup before traditional LLMs like BERT. We were definitely worried about accuracy and reliability of the medical advice then, and we had clinicians working closely to make sure the dialog trees were trustworthy. I work in aerospace medicine and aviation safety now, and I constantly encounter inadvisable use of LLMs and a lack of effective evaluation methods (especially for domain-specific LLMs). I appreciate the advisory notice in the README and the recommendation against using this in settings that may impact people. I sincerely hope that it's used ethically and responsibly.

ryandvm 3y ago

Sure, but we already have 250,000 medical deaths PER YEAR in the US due to medical errors (https://pubmed.ncbi.nlm.nih.gov/28186008/ https://pubmed.ncbi.nlm.nih.gov/28186008/). I don't think people should trust LLMs completely, but let's be real, they shouldn't trust humans completely either.

blipmusic 3y ago
Isn’t that whataboutism at its best? Those two things are completely unrelated.
- mannyv 3y ago
  No, it's showing that the risk of errors exists even without AI. AI doesn't necessarily make that risk higher or lower a priori. Plus if you knew how much of current medical practice exists without evidence you wouldn't be worrying about AI.
  - blipmusic 3y ago
    Maybe it’s ok to worry about both? Not trusting ”arbitrary thing A” does not logically make ”arbitrary thing B” more trustworthy. I do realise that these models intend to (incrementally) represent collective knowledge and may get there in the future. But if you worry about A, why not worry about B which is based on A?
  - not2b 3y ago
    You seem to be assuming, without any evidence at all, that LLMs giving medical advice are likely to be roughly equivalent in accuracy to doctors who are actually examining the patient and not just processing language, just because you are aware that medical mistakes are common.
    - famouswaffles 3y ago
      https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10425828/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10425828/ Use of GPT-4 to Analyze Medical Records of Patients With Extensive Investigations and Delayed Diagnosis "Six patients 65 years or older (2 women and 4 men) were included in the analysis. The accuracy of the primary diagnoses made by GPT-4, clinicians, and Isabel DDx Companion was 4 of 6 patients (66.7%), 2 of 6 patients (33.3%), and 0 patients, respectively. If including differential diagnoses, the accuracy was 5 of 6 (83.3%) for GPT-4, 3 of 6 (50.0%) for clinicians, and 2 of 6 (33.3%) for Isabel DDx Companion"

davidjade 3y ago

Here’s a recent (yesterday) example of a benefit though. I tried unsuccessfully to search for an ECG analysis term (EAR or EA Run) using Google, DDG, etc. There was no magic set of quoting, search terms, etc. that could explain what those terms were. Ear is just too common for a word. ChatGPT however was able to take the context of the question I had (an ECG analysis) and lead me to the answer right away of what EAR meant. I wasn’t seeking medical advice though, just a better search engine with context. So there are clearly benefits here too.

nhinck2 3y ago
Ectopic Atrial Rhythm?
- TaylorAlexander 3y ago
  Yeah I took the person's comment, snipped "ECG analysis term EAR meaning" from it and popped that in to google, found a page "ECG Glossary, Terms, & Terminology List" and under the E section it has "Ectopic Atrial Rhythm". That said if its correct, LLMs are less work for this kind of thing. But often when people explain to another person what they are looking for, as done in their comment, they will do a better job explaining what they need than when they are in their own head trying to google it. Which is why I just snipped their words to search for it.

deleted 3y ago

[deleted]

BrandoElFollito 3y ago

On the other hand, your MD is going to look for the obvious, or statistically relevant, or currently prominent disease. But they could be presented 99% probability for flu, 1% or wazalla, and that testing for wazalla means pinching your ear tout may actually be correctly diagnosed sometimes. It is not that MDs are incompetent, it is just that when wazalla was briefly mentioned during their studies, they happened to be in the toilets and missed it. Flu was mentioned 76 times because it is common. Disclaimer: I know medicine from "House, MD" but also witnessed a miraculous diagnosis on my father just because his MD happened to read an obscure article (for the story, he was diagnosed with a worm-induced illness that happened one or twice a year in France in the 80's. The worm was from a beach in Brazil, and my dad never travelled to Americas. He was kindly asked to provide a sample of blood to help research in France, which he did. Finally the drug to heal him was available in one pharmacy in Paris and in Lyon. We expected a hefty cost (though it is all covered in France), it costed 5 franks or so. But we were told with my brother to keep an eye on him as he may become delusional and try to jump through the window. The poor man cold hardly blink before we were on him:) Ah, and the pills were 2cm wide, looked like they were for an elephant. And he had 5 or so to swallow)

firebot 3y ago

What's to be terrified about? Humans also hallucinate. Doctors are terrible at their jobs.

meroes 3y ago
If doctors are terrible at their jobs so are programmers. And we should be terrified just as well at their AI creations then.
- firebot 3y ago
  That's a fair point. Most programmers are terrible. As a lead programmer I've dealt with countless "programmers" that can not do their job without me spoon-feeding them code.

infecto 3y ago

I am personally excited for the possibilities. Nobody should be using a LLM without verifying. Will some people do it? Of course, I remember that court case where the lawyer used ChatGPT and it made up cases. If someone is going to make that mistake, there were other mistakes happening, not just using a LLM. On the positive note. LLMs offer the chance to potentially do much better diagnosing on hard to figure out cases.

renewiltord 3y ago

It's true. Only people like me should be allowed access to LLMs. Folks like you should be protected. Equivalent to accredited investor, there should be a tier of "knowledgeable normal person" who is allowed to do whatever.

brianjking 3y ago

In fairness, many doctors have terrible advice and make mistakes often. This is why malpractice insurance exists.

Exoristos 3y ago

Most deaths are already caused by the medical establishment.

ekianjo 3y ago

You heard about the bell curve concept? Chances are about half the doctors you see are at the lower part of the curve. Which means they are borderline or completely incompetent at what they do. I'd take my chances with a "properly trained" AI any day. Problem is, most medical corpus is full of bogus studies that have never been replicated, so it might be close to junk at this stage. > It will lead to deaths, regular doctors kill people everyday and get away with it because you accept the risks. What's different?