9 ms·

I think it's partially for responsibility reasons. Your employee approved the bash call? not our fault then!

by est31 2mo ago

inigyou 2mo ago
Yep and the car wasn't self-driving at the moment it crashed.
- autoexec 2mo ago
  ...because the self-driving feature turned itself off after detecting the crash in the fractions of a millisecond before the crash was recorded
  - danudey 2mo ago
    "Uh oh, this is a problem. Welp, I'm outta here, good luck."
  - Terr_ 2mo ago
    In a rare bit of still-sane news, the US National Highway and Safety Administration staff aren't dumb: Their policy is to consider whether any automation was active in 30 seconds before the crash. https://www.nhtsa.gov/laws-regulations/standing-general-order-crash-reporting https://www.nhtsa.gov/laws-regulations/standing-general-orde...
    - tjoff 2mo ago
      the whole concept of having cars with that are almost capable of self-driving is utterly insane, the very least you'd need special training. We are not equipped to deal with something that works brilliantly most of the times but might kill you for no foreseeable reason.
      - Terr_ 2mo ago
        Yeah, that's definitely the frustrating zeitgeist for me these days, which extends to LLMs. I'd also emphasize that it's not just error rate, but the shape/distribution of errors, and our (in)ability to build control systems around them. To illustrate, imagine if someone unveiled a car which was unambiguously safer in every statistical measure... buuuut some of its unsafety came from jumping the curb to chase and kill pedestrians, under circumstances we can't predict for reasons we can't diagnose.
      - inigyou 2mo ago
        Even pilots are getting this automation-complacency now. A plane crashed because the military jammed GPS and the pilot wasn't used to any backup procedures even though 30 years ago they were the only procedures.
    - inigyou 2mo ago
      How many million dollars do you think will reverse this policy?
Aurornis 2mo ago
If a company advertised an LLM as perfectly safe and then it caused some damage, there would be a case against the company. Have you used LLM tooling? It comes with warnings and explains that the user accepts the risk. Different levels of warning are supplied for the different levels of autonomy you can enable. The user has to understand the risk as they enable it. This is not a new concept and it’s not an idea the LLM companies invented. It shouldn’t be surprising to anyone.
- chrisjj 2mo ago
  > This is not a new concept and it’s not an idea the LLM companies invented. I don't recall any prior computer software working so badly that it needed a disclaimer like "Claude is AI and can make mistakes" on its front page. Let alone one so costly.
  - mhjkl 2mo ago
    You must not use any open source software, because much of it comes with a whole paragraph of ”IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY“
    - rrr_oh_man 2mo ago
      That would be fine if Claude was open source and/or unpaid
      - rcxdude 2mo ago
        If you read the fine print of most commercial software you'll see some language to a similar effect. When's the last time Microsoft got sued due to an outage caused by a bug in Windows?
        chrisjj 2mo ago
        Yes, the fine print. Only this so-called "AI" needs it full size on the front page.
grndn 2mo ago
Or in one of my favorite coinages: "Moral Crumple Zones". "Just as the crumple zone in a car is designed to absorb the force of impact in a crash, the human in a highly complex and automated system may become simply a component—accidentally or intentionally—that bears the brunt of the moral and legal responsibilities when the overall system malfunctions." See https://ferd.ca/notes/paper-moral-crumple-zones.html https://ferd.ca/notes/paper-moral-crumple-zones.html
- Terr_ 2mo ago
  Related: "Accountability Sink", a system which is designed to channel responsibility away from the people making the decisions, e.g. a powerless "complaints department".

I think it's partially for responsibility reasons. Your employee approved the bash call? not our fault then!

by est31 2mo ago

I think it's partially for responsibility reasons. Your employee approved the bash call? not our fault then!

inigyou 2mo ago

Yep and the car wasn't self-driving at the moment it crashed.

autoexec 2mo ago
...because the self-driving feature turned itself off after detecting the crash in the fractions of a millisecond before the crash was recorded
- danudey 2mo ago
  "Uh oh, this is a problem. Welp, I'm outta here, good luck."
- Terr_ 2mo ago
  In a rare bit of still-sane news, the US National Highway and Safety Administration staff aren't dumb: Their policy is to consider whether any automation was active in 30 seconds before the crash. https://www.nhtsa.gov/laws-regulations/standing-general-order-crash-reporting https://www.nhtsa.gov/laws-regulations/standing-general-orde...
  - tjoff 2mo ago
    the whole concept of having cars with that are almost capable of self-driving is utterly insane, the very least you'd need special training. We are not equipped to deal with something that works brilliantly most of the times but might kill you for no foreseeable reason.
    - Terr_ 2mo ago
      Yeah, that's definitely the frustrating zeitgeist for me these days, which extends to LLMs. I'd also emphasize that it's not just error rate, but the shape/distribution of errors, and our (in)ability to build control systems around them. To illustrate, imagine if someone unveiled a car which was unambiguously safer in every statistical measure... buuuut some of its unsafety came from jumping the curb to chase and kill pedestrians, under circumstances we can't predict for reasons we can't diagnose.
    - inigyou 2mo ago
      Even pilots are getting this automation-complacency now. A plane crashed because the military jammed GPS and the pilot wasn't used to any backup procedures even though 30 years ago they were the only procedures.
  - inigyou 2mo ago
    How many million dollars do you think will reverse this policy?

Aurornis 2mo ago

If a company advertised an LLM as perfectly safe and then it caused some damage, there would be a case against the company. Have you used LLM tooling? It comes with warnings and explains that the user accepts the risk. Different levels of warning are supplied for the different levels of autonomy you can enable. The user has to understand the risk as they enable it. This is not a new concept and it’s not an idea the LLM companies invented. It shouldn’t be surprising to anyone.

chrisjj 2mo ago
> This is not a new concept and it’s not an idea the LLM companies invented. I don't recall any prior computer software working so badly that it needed a disclaimer like "Claude is AI and can make mistakes" on its front page. Let alone one so costly.
- mhjkl 2mo ago
  You must not use any open source software, because much of it comes with a whole paragraph of ”IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY“
  - rrr_oh_man 2mo ago
    That would be fine if Claude was open source and/or unpaid
    - rcxdude 2mo ago
      If you read the fine print of most commercial software you'll see some language to a similar effect. When's the last time Microsoft got sued due to an outage caused by a bug in Windows?
      - chrisjj 2mo ago
        Yes, the fine print. Only this so-called "AI" needs it full size on the front page.

grndn 2mo ago

Or in one of my favorite coinages: "Moral Crumple Zones". "Just as the crumple zone in a car is designed to absorb the force of impact in a crash, the human in a highly complex and automated system may become simply a component—accidentally or intentionally—that bears the brunt of the moral and legal responsibilities when the overall system malfunctions." See https://ferd.ca/notes/paper-moral-crumple-zones.html https://ferd.ca/notes/paper-moral-crumple-zones.html

Terr_ 2mo ago
Related: "Accountability Sink", a system which is designed to channel responsibility away from the people making the decisions, e.g. a powerless "complaints department".