5 ms·

I know this is just an example, but I'm going to say this every time the subject comes up: since human lives are the cost of making the wrong decisions, AI can'

I know this is just an example, but I'm going to say this every time the subject comes up: since human lives are the cost of making the wrong decisions, AI can't be allowed to be a decision maker of any kind in healthcare. If my blood pressure crashes because my nurse is caring for 15 patients at once and didn't notice that the AI decided to pause my norepinephrine, I won't consider "the AI did it" to be any acceptable excuse.

zahlman 22d ago
I want to emphasize your point. As a basic matter of valuing humanity qua humanity, this holds even if the AI could be shown to make the decisions more accurately than a human could. In fact, especially in those circumstances, if we take the warnings of X-risk even slightly seriously.
RigelKentaurus 22d ago
Not to say that I don't agree with you to an extent, but you should see AI-delivered healthcare numbers to human-delivered and make it a fair comparison. From Google: "Annual Deaths: Over 250,000 people die each year in the U.S. due to medical errors made by healthcare professionals, including doctors and nurses. Ranking: This figure positions medical errors as the third leading cause of death in the United States, following heart disease and cancer." It is highly possible that in a few years, AI will be a lot safer in the majority of healthcare situations.
- ryandrake 22d ago
  The difference is that humans are at least supposed to be accountable for those 250,000 errors. Those humans can naturally care about doing a good job, or they can be motivated to care. They can fear consequences and take pride in avoiding mistakes. AI can do none of this, and cannot be held accountable for mistakes. I would take a human with a 10% mistake rate but who can be held accountable, over an AI with a 0.1% mistake rate that is totally unaccountable for its mistakes.
  - jurgenburgen 22d ago
    > I would take a human with a 10% mistake rate but who can be held accountable, over an AI with a 0.1% mistake rate that is totally unaccountable for its mistakes. You would take those odds? 100x more deaths? I agree that AI _companies_ should be accountable for the software they produce but I’m not convinced you would go for a 90% chance of survival if you can choose 99.9%.
    - ryandrake 22d ago
      I guess if you're dead, then it ultimately doesn't matter if someone can be held accountable, although it might matter to your next-of-kin. But, yea, in general, it would take a huge (1000X or 10,000X) difference in expected outcome for me to trust something that is unaccountable. If the AI company was actually held accountable for its AI's mistakes, I'd definitely change my statement.
      - bdangubic 22d ago
        AI is a tool just like any other tool. Holding “AI” accountable is like holding JetBrains accountable for the bugs I wrote with their tool than endangered lives of people
        pixl97 22d ago
        Heh, just wait till some sovereign AI gets loose on the internet.
        krapp 22d ago
        There is simply too much money and too many interests vested in the narrative (which many people already believe) that LLMs are thinking, feeling, sapient beings already beyond the threshold of AGI that deserve some status of legal personhood and liability. It is impossible to create the future that these people want to create - one in which AI comprises the new legal class of nobility in an age of technofeudalism - if AI remains just a tool.
- bluerooibos 22d ago
  This. It's the exact same argument that was being made (maybe still is) against self driving cars. To be honest, I'm looking forward to more AI in healthcare if it can increase diagnosis success rates and improve treatment outcomes etc.
  - slaw3 22d ago
    I think in 20 years we will have ironed this out and will be able to prolong meaningful, high functioning life, using less resources than we do today. The next decade will be painful. We are in for a lot of surprises along the way
- NoDodgeQuestion 22d ago
  https://www.mcgill.ca/oss/article/critical-thinking-health/medical-error-not-third-leading-cause-death https://www.mcgill.ca/oss/article/critical-thinking-health/m... "Taking death statistics from a sample of Medicare patients and extrapolating it to all hospitalized patients is like turning apples into oranges" "Using studies that identify medical errors that were followed by death to declare that these medical errors necessarily caused these deaths is not fair. What these studies do not take into account is how long these patients would have lived had they received optimal medical care."
tengbretson 22d ago
> AI can't be allowed to be a decision maker of any kind in healthcare. It will, except it will be snuck in through the back door. It won't be some kind of robo-doctor making care decisions, it will come in the form of insurance eligibility steering the approved care plans, which absolutely will be decided by AI. The doctors will remain to act as liability sinks, but functionally they will probably join the ranks of the meat proxies.
- trhway 22d ago
  I see that already in veterinary medicine with the non-DVM office/practice managers (or whatever title that guy lurking around in the office may have) - in the vet practices belonging to large networks the DVMs and nurses constantly looking at those managers for approval.
  - pessimizer 22d ago
    Vets fret as private equity snaps up clinics, pet care companies https://stateline.org/2024/03/29/vets-fret-as-private-equity-snaps-up-clinics-pet-care-companies/ https://stateline.org/2024/03/29/vets-fret-as-private-equity...
marssaxman 22d ago
Why wouldn't they just blame the nurse anyway? The law always puts somebody on the hook. The nurse's job will be to do what the AI says and take the blame when it fails.
- 0xdeadbeefbabe 22d ago
  At least there will be more nurses to take the blame.
  - ElProlactin 22d ago
    And electricians!
panarky 22d ago
> human lives are the cost of making the wrong decisions ... True. > ... AI can't be allowed to be a decision maker Non sequitur. Part 2 only follows from the Part 1 if you can demonstrate that AI makes more errors, or more severe errors, than humans in the identical situation. And for those situations, I agree with you. But there are already many situations where the AI is making fewer errors than humans, and for those situations, it would be unethical to exclude AI (see Part 1).
- engeljohnb 22d ago
  Since AI can't be held liable for mistakes, it's immaterial whether the AI makes fewer mistakes than humans.
  - GlacierFox 22d ago
    This isn't going to compute with the AI evangelists on here.
  - RigelKentaurus 22d ago
    Not true. The company deploying or using the AI (not the frontier AI lab building it) needs to make sure that the AI works. Just like anything else. Not sure why this is complicated.
    - pessimizer 22d ago
      > The company deploying or using the AI (not the frontier AI lab building it) needs to make sure that the AI works. Ah, they need to. Thanks for clearing this up.
  - marcus_holmes 22d ago
    Why can't the AI be held liable? If I pay a company to do something, and that company uses an AI to do the thing, and the AI messes it up, why isn't the company liable?
  - tekne 22d ago
    Surely the only rational reason to care about liability is you get paid in the bad outcome. Let's assume you get paid 0$ for AI mistakes -- no liability at all. Then your EV is roughly: (AI accuracy - human accuracy) * (cost of accident) - (1 - human accuracy) * (human payout) Now I don't know about you, but even with pretty big numbers on the human payout scale and relatively small differences, I would not accept 10 million (presumably paid to my next of kin) to die. Does lead to weird situations where I may prefer humans for minor medical procedures where I can take the risk -- but only in edge cases. Or is there some weird punishment thing going on here?
sdevonoes 22d ago
> “the AI did it" to be any acceptable excuse. It will be depending on how much you paid. My bet is that in the near future normal people (like most of us) will be attended by AI in pretty much everywhere: the bank, the hospital, shops, hotels, etc. You will need to pay extra premium to be attended by a human being. Actually that’s what’s been happening to the ultra rich since a very long time ago (e.g. a millionaire doesn’t book an appointment by using their phones, they ask their assistant to book for them; they don’t go online shopping, they get people do the shopping for them, etc)
pjmorris 22d ago
"A computer can never be held accountable; therefore, a computer must never make a management decision." - IBM, 1979 I'd say society's, or maybe executive's, viewpoint has shifted on this.
luka2233 22d ago
There was quite a long debate about "Hallucination Defense" here: https://news.ycombinator.com/item?id=46815527 https://news.ycombinator.com/item?id=46815527
PaulDavisThe1st 22d ago
Excuse for who? The nurse? The hospital administration? The hospital owners? The insurance company?
- pessimizer 22d ago
  Your comment is how blame dissolving into the ether looks.
  - PaulDavisThe1st 21d ago
    Then I suppose your reply should have said "all of them" ?
throwaway13337 22d ago
Replace AI with 'company' and you realize that we're living in that world already. There's a reason it's called a Limited Liability Company. The whole point is to separate the company's liability from the individual that participates in it. For example, corporate financial liabilities of a company that is bankrupt. You would not sue the nurse that makes a mistake. You would sue the company she works for. And if that company is bankrupt, well, that's it. There are rare instances where this isn't true, but generally it is. Whether this is ethical or not, it's the system that we already have.
- deleted 22d ago
  [deleted]
- daveguy 22d ago
  I don't think they meant the nurse would be responsible. The LLC would and should be fucked in that case.
  - wizardforhire 22d ago
    And I think the apprehension can be summed with “good luck with that”
    - davidgay 22d ago
      I could've sworn that malpractice insurance cost is viewed as one of the driver's of high health care costs, and that it's expensive precisely because medical lawsuit payouts are large...
      - shermantanktop 22d ago
        Insurance and HMO CEOs salaries come from somewhere, and it’s not malpractice judgments.
  - throwaway13337 22d ago
    Whether the company employs a human nurse or an AI, the company and not a human is indeed considered the responsible party. The LLC might be fucked but it is not a flesh-and-blood person. There is no person responsible. Just a legal entity. So nothing is changed. I think we're in agreement?
deleted 22d ago
[deleted]
slaw3 22d ago
> If my blood pressure crashes because my nurse is caring for 15 patients at once and didn't notice that the AI decided to pause my norepinephrine, I won't consider "the AI did it" to be any acceptable excuse. While not quite as direct as automatically stopping vasopressors, using proprietary prediction models touted by vendors to be safe has backfired in a major way before: https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2781307 https://jamanetwork.com/journals/jamainternalmedicine/fullar... https://www.statnews.com/2021/09/27/epic-sepsis-algorithm-antibiotics-model https://www.statnews.com/2021/09/27/epic-sepsis-algorithm-an...
bloppe 22d ago
This is the same faulty standard you constantly hear applied to self-driving cars, where human lives are also often at stake. You don't need the AI to be perfect. You just need it to be substantially better than the average human (which self-driving cars are). Your nurse who's caring for 15 patients at once might not make perfect decisions either. That doesn't categorically disqualify humans from delivering health care

zahlman 22d ago

I want to emphasize your point. As a basic matter of valuing humanity qua humanity, this holds even if the AI could be shown to make the decisions more accurately than a human could. In fact, especially in those circumstances, if we take the warnings of X-risk even slightly seriously.

RigelKentaurus 22d ago

Not to say that I don't agree with you to an extent, but you should see AI-delivered healthcare numbers to human-delivered and make it a fair comparison. From Google: "Annual Deaths: Over 250,000 people die each year in the U.S. due to medical errors made by healthcare professionals, including doctors and nurses. Ranking: This figure positions medical errors as the third leading cause of death in the United States, following heart disease and cancer." It is highly possible that in a few years, AI will be a lot safer in the majority of healthcare situations.

ryandrake 22d ago
The difference is that humans are at least supposed to be accountable for those 250,000 errors. Those humans can naturally care about doing a good job, or they can be motivated to care. They can fear consequences and take pride in avoiding mistakes. AI can do none of this, and cannot be held accountable for mistakes. I would take a human with a 10% mistake rate but who can be held accountable, over an AI with a 0.1% mistake rate that is totally unaccountable for its mistakes.
- jurgenburgen 22d ago
  > I would take a human with a 10% mistake rate but who can be held accountable, over an AI with a 0.1% mistake rate that is totally unaccountable for its mistakes. You would take those odds? 100x more deaths? I agree that AI _companies_ should be accountable for the software they produce but I’m not convinced you would go for a 90% chance of survival if you can choose 99.9%.
  - ryandrake 22d ago
    I guess if you're dead, then it ultimately doesn't matter if someone can be held accountable, although it might matter to your next-of-kin. But, yea, in general, it would take a huge (1000X or 10,000X) difference in expected outcome for me to trust something that is unaccountable. If the AI company was actually held accountable for its AI's mistakes, I'd definitely change my statement.
    - bdangubic 22d ago
      AI is a tool just like any other tool. Holding “AI” accountable is like holding JetBrains accountable for the bugs I wrote with their tool than endangered lives of people
      - pixl97 22d ago
        Heh, just wait till some sovereign AI gets loose on the internet.
      - krapp 22d ago
        There is simply too much money and too many interests vested in the narrative (which many people already believe) that LLMs are thinking, feeling, sapient beings already beyond the threshold of AGI that deserve some status of legal personhood and liability. It is impossible to create the future that these people want to create - one in which AI comprises the new legal class of nobility in an age of technofeudalism - if AI remains just a tool.
bluerooibos 22d ago
This. It's the exact same argument that was being made (maybe still is) against self driving cars. To be honest, I'm looking forward to more AI in healthcare if it can increase diagnosis success rates and improve treatment outcomes etc.
- slaw3 22d ago
  I think in 20 years we will have ironed this out and will be able to prolong meaningful, high functioning life, using less resources than we do today. The next decade will be painful. We are in for a lot of surprises along the way
NoDodgeQuestion 22d ago
https://www.mcgill.ca/oss/article/critical-thinking-health/medical-error-not-third-leading-cause-death https://www.mcgill.ca/oss/article/critical-thinking-health/m... "Taking death statistics from a sample of Medicare patients and extrapolating it to all hospitalized patients is like turning apples into oranges" "Using studies that identify medical errors that were followed by death to declare that these medical errors necessarily caused these deaths is not fair. What these studies do not take into account is how long these patients would have lived had they received optimal medical care."

tengbretson 22d ago

> AI can't be allowed to be a decision maker of any kind in healthcare. It will, except it will be snuck in through the back door. It won't be some kind of robo-doctor making care decisions, it will come in the form of insurance eligibility steering the approved care plans, which absolutely will be decided by AI. The doctors will remain to act as liability sinks, but functionally they will probably join the ranks of the meat proxies.

trhway 22d ago
I see that already in veterinary medicine with the non-DVM office/practice managers (or whatever title that guy lurking around in the office may have) - in the vet practices belonging to large networks the DVMs and nurses constantly looking at those managers for approval.
- pessimizer 22d ago
  Vets fret as private equity snaps up clinics, pet care companies https://stateline.org/2024/03/29/vets-fret-as-private-equity-snaps-up-clinics-pet-care-companies/ https://stateline.org/2024/03/29/vets-fret-as-private-equity...

marssaxman 22d ago

Why wouldn't they just blame the nurse anyway? The law always puts somebody on the hook. The nurse's job will be to do what the AI says and take the blame when it fails.

0xdeadbeefbabe 22d ago
At least there will be more nurses to take the blame.
- ElProlactin 22d ago
  And electricians!

panarky 22d ago

> human lives are the cost of making the wrong decisions ... True. > ... AI can't be allowed to be a decision maker Non sequitur. Part 2 only follows from the Part 1 if you can demonstrate that AI makes more errors, or more severe errors, than humans in the identical situation. And for those situations, I agree with you. But there are already many situations where the AI is making fewer errors than humans, and for those situations, it would be unethical to exclude AI (see Part 1).

engeljohnb 22d ago
Since AI can't be held liable for mistakes, it's immaterial whether the AI makes fewer mistakes than humans.
- GlacierFox 22d ago
  This isn't going to compute with the AI evangelists on here.
- RigelKentaurus 22d ago
  Not true. The company deploying or using the AI (not the frontier AI lab building it) needs to make sure that the AI works. Just like anything else. Not sure why this is complicated.
  - pessimizer 22d ago
    > The company deploying or using the AI (not the frontier AI lab building it) needs to make sure that the AI works. Ah, they need to. Thanks for clearing this up.
- marcus_holmes 22d ago
  Why can't the AI be held liable? If I pay a company to do something, and that company uses an AI to do the thing, and the AI messes it up, why isn't the company liable?
- tekne 22d ago
  Surely the only rational reason to care about liability is you get paid in the bad outcome. Let's assume you get paid 0$ for AI mistakes -- no liability at all. Then your EV is roughly: (AI accuracy - human accuracy) * (cost of accident) - (1 - human accuracy) * (human payout) Now I don't know about you, but even with pretty big numbers on the human payout scale and relatively small differences, I would not accept 10 million (presumably paid to my next of kin) to die. Does lead to weird situations where I may prefer humans for minor medical procedures where I can take the risk -- but only in edge cases. Or is there some weird punishment thing going on here?

sdevonoes 22d ago

> “the AI did it" to be any acceptable excuse. It will be depending on how much you paid. My bet is that in the near future normal people (like most of us) will be attended by AI in pretty much everywhere: the bank, the hospital, shops, hotels, etc. You will need to pay extra premium to be attended by a human being. Actually that’s what’s been happening to the ultra rich since a very long time ago (e.g. a millionaire doesn’t book an appointment by using their phones, they ask their assistant to book for them; they don’t go online shopping, they get people do the shopping for them, etc)

pjmorris 22d ago

"A computer can never be held accountable; therefore, a computer must never make a management decision." - IBM, 1979 I'd say society's, or maybe executive's, viewpoint has shifted on this.

luka2233 22d ago

There was quite a long debate about "Hallucination Defense" here: https://news.ycombinator.com/item?id=46815527 https://news.ycombinator.com/item?id=46815527

PaulDavisThe1st 22d ago

Excuse for who? The nurse? The hospital administration? The hospital owners? The insurance company?

pessimizer 22d ago
Your comment is how blame dissolving into the ether looks.
- PaulDavisThe1st 21d ago
  Then I suppose your reply should have said "all of them" ?

throwaway13337 22d ago

Replace AI with 'company' and you realize that we're living in that world already. There's a reason it's called a Limited Liability Company. The whole point is to separate the company's liability from the individual that participates in it. For example, corporate financial liabilities of a company that is bankrupt. You would not sue the nurse that makes a mistake. You would sue the company she works for. And if that company is bankrupt, well, that's it. There are rare instances where this isn't true, but generally it is. Whether this is ethical or not, it's the system that we already have.

deleted 22d ago
[deleted]
daveguy 22d ago
I don't think they meant the nurse would be responsible. The LLC would and should be fucked in that case.
- wizardforhire 22d ago
  And I think the apprehension can be summed with “good luck with that”
  - davidgay 22d ago
    I could've sworn that malpractice insurance cost is viewed as one of the driver's of high health care costs, and that it's expensive precisely because medical lawsuit payouts are large...
    - shermantanktop 22d ago
      Insurance and HMO CEOs salaries come from somewhere, and it’s not malpractice judgments.
- throwaway13337 22d ago
  Whether the company employs a human nurse or an AI, the company and not a human is indeed considered the responsible party. The LLC might be fucked but it is not a flesh-and-blood person. There is no person responsible. Just a legal entity. So nothing is changed. I think we're in agreement?

deleted 22d ago

[deleted]

slaw3 22d ago

> If my blood pressure crashes because my nurse is caring for 15 patients at once and didn't notice that the AI decided to pause my norepinephrine, I won't consider "the AI did it" to be any acceptable excuse. While not quite as direct as automatically stopping vasopressors, using proprietary prediction models touted by vendors to be safe has backfired in a major way before: https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2781307 https://jamanetwork.com/journals/jamainternalmedicine/fullar... https://www.statnews.com/2021/09/27/epic-sepsis-algorithm-antibiotics-model https://www.statnews.com/2021/09/27/epic-sepsis-algorithm-an...

bloppe 22d ago

This is the same faulty standard you constantly hear applied to self-driving cars, where human lives are also often at stake. You don't need the AI to be perfect. You just need it to be substantially better than the average human (which self-driving cars are). Your nurse who's caring for 15 patients at once might not make perfect decisions either. That doesn't categorically disqualify humans from delivering health care