6 ms·

Good lord. The idea that a system that produces pseudo-random output without any semantic understanding can be relied on to give accurate health-related informa

by imiric 1y ago

Good lord. The idea that a system that produces pseudo-random output without any semantic understanding can be relied on to give accurate health-related information is deeply flawed and troubling. It's one thing to use these things for finding patterns in data, for entertainment purposes, and for producing nonsensical code a human has to fix, but entirely different to rely on them for health diagnosis or advice. I shudder at the thought that a medical practitioner I go to will parrot whatever an LLM told them. This insanity needs to be regulated yesterday.

FlamingMoe 1y ago
Good lord. The idea that a sleep‑deprived, bias‑prone mammal can be relied on to give accurate health‑related information is deeply flawed and troubling. It’s one thing to seek a human for empathy, small talk, or a routine vaccine, but entirely different to assume an overworked doctor will consistently out‑diagnose a modern data tool. I shudder at the thought that the practitioner I visit might simply repeat whatever a drug‑rep lunch pitch or outdated guideline left in their memory.
- stavros 1y ago
  This insanity needs to be regulated yesterday.
  - imiric 1y ago
    Thankfully, it is. :)
    - hiatus 1y ago
      Are there regulations regarding doctor's working hours similar to say, truck drivers? I found this [1] article which states doctors are limited to 80 hours a week whereas truck drivers are limited to 70 hours in a 7 day period. [2] [1] https://psnet.ahrq.gov/primer/duty-hours-and-patient-safety https://psnet.ahrq.gov/primer/duty-hours-and-patient-safety [2] https://www.fmcsa.dot.gov/sites/fmcsa.dot.gov/files/docs/Drivers%20Guide%20to%20HOS%202015_508.pdf https://www.fmcsa.dot.gov/sites/fmcsa.dot.gov/files/docs/Dri...
      - n8henrie 1y ago
        Those restrictions are for residents (trainees). They were also found not to improve patient outcomes (possibly due to increased number of handoffs, which are highly error prone).
        hiatus 1y ago
        Do you think it makes sense that doctors can work longer hours than truck drivers?
        n8henrie 1y ago
        If the available evidence does not show an improvement in patient outcomes with shorter hours, and if longer hours are worked willingly and enable them to treat more patients (iso widespread physician shortages), yes I do. Turning the question around, if mandating shorter hours for physicians were found to result in higher rates of significant patient harm, would you be in favor of those restrictions?
      - trollbridge 1y ago
        Doctors are in high demand, and don’t willingly work at places the demand long hours. Primary care doctors generally aren’t residents.
creata 1y ago
Can you state more clearly why it's deeply flawed? Because while LLMs obviously have massive limitations, so do humans, and it's not entirely clear to me that some synthesis of the two can't produce much better results than either on its own.
- baby_souffle 1y ago
  > ... some synthesis of the two can't produce much better results than either on its own. In theory, I agree with you. The world "some" is doing a lot of heavy lifting there though. I only hope that whatever definition of some emerges, it's not a horribly flawed one.
  - imiric 1y ago
    That's my hope as well. A responsible doctor that uses technology critically and knows its limitations is fine with me. My concern is that it's more likely that whatever AI says will be taken for granted, overruling any human judgment, because they're supposed to be expert systems that scored highly on some made-up benchmark produced by the same company that trained the model. We see this today with AI-generated content on the web, and a flood of sloppily put together software produced by people who swear that AI is making them more productive. There's little interest in judging the output, and a lot of interest in lazy cash grabs. There are no guardrails in place in the healthcare industry AFAIA to prevent the same happening there, which is a scary thought.
- smusamashah 1y ago
  You would not hire or go to a a mentally unstable doctor. If a doc hallucinates or makes a mistake, it will be rare. They can self introspect and work on it if they realise there mistake even. AI is NOT same. It has no self reflection, hallucination isn't rare. AI is not mentally stable in any sense.
senordevnyc 1y ago
Whether you (or anyone else) actually understands something is opaque to me. For all I know, your comment was left by an AI. There is no magic wand we can wave to separate out the people who actually understand something (whatever that means), from the people who just appear to. The best we can do is test a human's mastery of a subject to estimate how well they actually know and understand that topic. Which is exactly what OpenAI is doing here. What I care about is the results. If the "grade" is 10%, then I don't want to rely on it, whether it's a human or an AI. If it's 95%, then I feel fine about relying on it. Especially since I suspect that very soon, most doctors would not score as well on a benchmark like this as the SOTA models.
- imiric 1y ago
  > What I care about is the results. If the "grade" is 10%, then I don't want to rely on it, whether it's a human or an AI. So you judge mastery of a subject by some arbitrary grade? Produced, incidentally, by the same company that trained the model. But I'm sure they wouldn't cheat... I also care about the results, but I judge it based on direct experience. IME the current SOTA models can't be relied on to accurately produce working code most of the time, yet I'm expected to trust they can produce accurate healthcare diagnosis and advice? Give me a break. AI companies have sold us a fantasy that a neat computer trick, which can still be very useful for society, will somehow lead to AGI. Now that we're a decade into this hype cycle and investors are getting antsy, they're doubling down by anthropomorphizing the technology and selling us "chain of thought" and "reasoning", as if making everyone say these things will somehow magically produce intelligence. What's really irresponsible is that the public and governments are buying into it, which is pushing the tech into industries that can actually harm people. Safety is not just about trusting some arbitrary grade.
  - senordevnyc 1y ago
    IME the current SOTA models can't be relied on to accurately produce working code most of the time My experience is the opposite.
  - famouswaffles 1y ago

by imiric 1y ago

FlamingMoe 1y ago

Good lord. The idea that a sleep‑deprived, bias‑prone mammal can be relied on to give accurate health‑related information is deeply flawed and troubling. It’s one thing to seek a human for empathy, small talk, or a routine vaccine, but entirely different to assume an overworked doctor will consistently out‑diagnose a modern data tool. I shudder at the thought that the practitioner I visit might simply repeat whatever a drug‑rep lunch pitch or outdated guideline left in their memory.

stavros 1y ago
This insanity needs to be regulated yesterday.
- imiric 1y ago
  Thankfully, it is. :)
  - hiatus 1y ago
    Are there regulations regarding doctor's working hours similar to say, truck drivers? I found this [1] article which states doctors are limited to 80 hours a week whereas truck drivers are limited to 70 hours in a 7 day period. [2] [1] https://psnet.ahrq.gov/primer/duty-hours-and-patient-safety https://psnet.ahrq.gov/primer/duty-hours-and-patient-safety [2] https://www.fmcsa.dot.gov/sites/fmcsa.dot.gov/files/docs/Drivers%20Guide%20to%20HOS%202015_508.pdf https://www.fmcsa.dot.gov/sites/fmcsa.dot.gov/files/docs/Dri...
    - n8henrie 1y ago
      Those restrictions are for residents (trainees). They were also found not to improve patient outcomes (possibly due to increased number of handoffs, which are highly error prone).
      - hiatus 1y ago
        Do you think it makes sense that doctors can work longer hours than truck drivers?
        n8henrie 1y ago
        If the available evidence does not show an improvement in patient outcomes with shorter hours, and if longer hours are worked willingly and enable them to treat more patients (iso widespread physician shortages), yes I do. Turning the question around, if mandating shorter hours for physicians were found to result in higher rates of significant patient harm, would you be in favor of those restrictions?
    - trollbridge 1y ago
      Doctors are in high demand, and don’t willingly work at places the demand long hours. Primary care doctors generally aren’t residents.

creata 1y ago

Can you state more clearly why it's deeply flawed? Because while LLMs obviously have massive limitations, so do humans, and it's not entirely clear to me that some synthesis of the two can't produce much better results than either on its own.

baby_souffle 1y ago
> ... some synthesis of the two can't produce much better results than either on its own. In theory, I agree with you. The world "some" is doing a lot of heavy lifting there though. I only hope that whatever definition of some emerges, it's not a horribly flawed one.
- imiric 1y ago
  That's my hope as well. A responsible doctor that uses technology critically and knows its limitations is fine with me. My concern is that it's more likely that whatever AI says will be taken for granted, overruling any human judgment, because they're supposed to be expert systems that scored highly on some made-up benchmark produced by the same company that trained the model. We see this today with AI-generated content on the web, and a flood of sloppily put together software produced by people who swear that AI is making them more productive. There's little interest in judging the output, and a lot of interest in lazy cash grabs. There are no guardrails in place in the healthcare industry AFAIA to prevent the same happening there, which is a scary thought.
smusamashah 1y ago
You would not hire or go to a a mentally unstable doctor. If a doc hallucinates or makes a mistake, it will be rare. They can self introspect and work on it if they realise there mistake even. AI is NOT same. It has no self reflection, hallucination isn't rare. AI is not mentally stable in any sense.

senordevnyc 1y ago

Whether you (or anyone else) actually understands something is opaque to me. For all I know, your comment was left by an AI. There is no magic wand we can wave to separate out the people who actually understand something (whatever that means), from the people who just appear to. The best we can do is test a human's mastery of a subject to estimate how well they actually know and understand that topic. Which is exactly what OpenAI is doing here. What I care about is the results. If the "grade" is 10%, then I don't want to rely on it, whether it's a human or an AI. If it's 95%, then I feel fine about relying on it. Especially since I suspect that very soon, most doctors would not score as well on a benchmark like this as the SOTA models.

imiric 1y ago
> What I care about is the results. If the "grade" is 10%, then I don't want to rely on it, whether it's a human or an AI. So you judge mastery of a subject by some arbitrary grade? Produced, incidentally, by the same company that trained the model. But I'm sure they wouldn't cheat... I also care about the results, but I judge it based on direct experience. IME the current SOTA models can't be relied on to accurately produce working code most of the time, yet I'm expected to trust they can produce accurate healthcare diagnosis and advice? Give me a break. AI companies have sold us a fantasy that a neat computer trick, which can still be very useful for society, will somehow lead to AGI. Now that we're a decade into this hype cycle and investors are getting antsy, they're doubling down by anthropomorphizing the technology and selling us "chain of thought" and "reasoning", as if making everyone say these things will somehow magically produce intelligence. What's really irresponsible is that the public and governments are buying into it, which is pushing the tech into industries that can actually harm people. Safety is not just about trusting some arbitrary grade.
- senordevnyc 1y ago
  IME the current SOTA models can't be relied on to accurately produce working code most of the time My experience is the opposite.
- famouswaffles 1y ago