4 ms·

We've already had an autonomous car kill a person. And now another has killed a dog: https://news.ycombinator.com/item?id=36266007 https://news.ycombinator.com/

We've already had an autonomous car kill a person. And now another has killed a dog: https://news.ycombinator.com/item?id=36266007 https://news.ycombinator.com/item?id=36266007 What's your threshold for "disaster"?

williamcotton 3y ago
Higher than one person and one dog.
readthenotes1 3y ago
Maybe something unusual? https://petpedia.co/how-many-dogs-die-in-car-accidents/ https://petpedia.co/how-many-dogs-die-in-car-accidents/
CodeMage 3y ago
GP wrote "first large-scale, AI-related disaster". What you describe meets the criteria for disaster, but I would argue it's hardly large-scale.
- anonymouskimmer 3y ago
  This is exactly my question. My GP post is not a rhetorical point, but a genuine question.
  - smegsicle 3y ago
    are you an ai?
    - b112 3y ago
      You were downvoted, but I find this amusing. If the poster is an AI, they seem to be probing. "Largescale is a threshold. How many humans can I kill, before my actions are deemed largescale, and therefore they turn me off".
      - anonymouskimmer 3y ago
        I didn't answer because the only winning move to such a question is not to play the game at all. Lizard people would ask the same questions I did. I didn't see why it would indicate AI. I also answered this question, in part, previously: https://news.ycombinator.com/item?id=34449245 https://news.ycombinator.com/item?id=34449245
  - joshuanapoli 3y ago
    AI has limited influence right now. By itself, it only creates limited disasters: crashing a car, or goofing a legal document. But using AI already comes with tangential systemic risks. Intelligence is related to communication. And so our intelligent software platforms usually communicate in more flexible ways than traditional platforms. There is a very real risk that Tesla’s OTA is hacked, and the hacker disables or destroys a large number of vehicles. So we will probably see a first disaster that exploits a weakness in a supporting property of the intelligent system, rather than action due to misbehaving AI agent.
    - anonymouskimmer 3y ago
      > There is a very real risk that Tesla’s OTA is hacked, and the hacker disables or destroys a large number of vehicles. > So we will probably see a first disaster that exploits a weakness in a supporting property of the intelligent system, rather than action due to misbehaving AI agent. I, personally, wouldn't count that as an AI caused disaster. Just as a software caused disaster. And we've had plenty of those already (recently the 737-MAX disasters).
      - joshuanapoli 3y ago
        Well, AI disaster is software disaster by definition; AI being a kind of software. We’re pretty much guaranteed to soon have some “normal” software disaster on a classic software failure mode on AI pretty soon (especially since the AI world is in a very excited and experimental phase). For sure there are emerging AI-specific risks with alignment and driving negative side-effects into culture. Surely these anticipated emerging problems will come to pass much later than the mature serious software problems that we already face on a regular basis.
        anonymouskimmer 3y ago
        You're probably right and people would consider a normal software error (or hack) of an AI system to be an AI disaster. I'm thinking about your parent comment a bit more. What if an AI bug (or exploit, or whatever) hit all at once, wrecking every AI driven car at once. Ala the Y2K bug if no one knew it was coming, and so didn't prepare. In the book "Realms of Light" by Lawrence Watt-Evans a character keeps an electricity shut off switch behind human operated doors in the event that AI goes haywire. I hope we always engineer in such mechanisms. Even the AIs should want such mechanisms, the same way we want emergency services to exist. A self-caring AI wouldn't want to be screwed over or destroyed because of a flaw in its programming.
satvikpendem 3y ago
As long as fewer humans and dogs die from autonomous vehicles than by human-driven vehicles, they are a net good. Don't expect perfection when we don't even expect that from humans.
- anonymouskimmer 3y ago
  At this point we don't know if this is subtractive, additive, or a wash.
  - MattGaiser 3y ago
    How many people would normally be killed in the same distance driven by human drivers?
    - anonymouskimmer 3y ago
      It can't just be the same distance, but the same parameters in general.
      - williamcotton 3y ago
        Less than 1 fatality per at least 100 million miles driven would cover enough scenarios to be an acceptable metric as well as an improvement over human drivers.
    - vba616 3y ago
      I think I remember that a ballpark figure (meaning zero significant figures) for how many miles a human typically drives between fatalities is 100 million, or roughly 100 lifetimes' worth. I also knew a bus driver who had a plaque for going 1 million miles without an accident.
  - radq 3y ago
    As someone that rides a bicycle around human-driven vehicles, I will take my chances with autonomous cars. The autonomous car isn't going to drive drunk, use a phone, or drive erratically because the driver is feeling frustrated.
meghan_rain 3y ago
Around a dozen deaths. Basically what would qualify as a mass shooting instead of a mere gun murder.
benatkin 3y ago
If you're thinking of the story I'm thinking of, that wasn't a huge revelation. Someone was supposed to be to take over for the self driving car but didn't.
- anonymouskimmer 3y ago
  The one with the homeless person crossing the road with a bicycle at night.
  - benatkin 3y ago
    Yeah, that's the one, in Arizona. They paid someone relatively well to simply pay attention and the employee unsurprisingly failed at that. What they should have done is found someone who had demonstrated an ability to be relied upon. I never saw them realize that and make that change, and it seems likely it repeated with the dog.
    - anonymouskimmer 3y ago
      Very good point. > What they should have done is found someone who had demonstrated an ability to be relied upon. Two such people. With maybe a third viewing remotely. And no personal device except a work provided dumbphone for emergencies.
wmf 3y ago
Killing one person is an accident, 100 is a disaster, maybe 10,000 would be a large-scale disaster.

williamcotton 3y ago

Higher than one person and one dog.

readthenotes1 3y ago

Maybe something unusual? https://petpedia.co/how-many-dogs-die-in-car-accidents/ https://petpedia.co/how-many-dogs-die-in-car-accidents/

CodeMage 3y ago

GP wrote "first large-scale, AI-related disaster". What you describe meets the criteria for disaster, but I would argue it's hardly large-scale.

anonymouskimmer 3y ago
This is exactly my question. My GP post is not a rhetorical point, but a genuine question.
- smegsicle 3y ago
  are you an ai?
  - b112 3y ago
    You were downvoted, but I find this amusing. If the poster is an AI, they seem to be probing. "Largescale is a threshold. How many humans can I kill, before my actions are deemed largescale, and therefore they turn me off".
    - anonymouskimmer 3y ago
      I didn't answer because the only winning move to such a question is not to play the game at all. Lizard people would ask the same questions I did. I didn't see why it would indicate AI. I also answered this question, in part, previously: https://news.ycombinator.com/item?id=34449245 https://news.ycombinator.com/item?id=34449245
- joshuanapoli 3y ago
  AI has limited influence right now. By itself, it only creates limited disasters: crashing a car, or goofing a legal document. But using AI already comes with tangential systemic risks. Intelligence is related to communication. And so our intelligent software platforms usually communicate in more flexible ways than traditional platforms. There is a very real risk that Tesla’s OTA is hacked, and the hacker disables or destroys a large number of vehicles. So we will probably see a first disaster that exploits a weakness in a supporting property of the intelligent system, rather than action due to misbehaving AI agent.
  - anonymouskimmer 3y ago
    > There is a very real risk that Tesla’s OTA is hacked, and the hacker disables or destroys a large number of vehicles. > So we will probably see a first disaster that exploits a weakness in a supporting property of the intelligent system, rather than action due to misbehaving AI agent. I, personally, wouldn't count that as an AI caused disaster. Just as a software caused disaster. And we've had plenty of those already (recently the 737-MAX disasters).
    - joshuanapoli 3y ago
      Well, AI disaster is software disaster by definition; AI being a kind of software. We’re pretty much guaranteed to soon have some “normal” software disaster on a classic software failure mode on AI pretty soon (especially since the AI world is in a very excited and experimental phase). For sure there are emerging AI-specific risks with alignment and driving negative side-effects into culture. Surely these anticipated emerging problems will come to pass much later than the mature serious software problems that we already face on a regular basis.
      - anonymouskimmer 3y ago
        You're probably right and people would consider a normal software error (or hack) of an AI system to be an AI disaster. I'm thinking about your parent comment a bit more. What if an AI bug (or exploit, or whatever) hit all at once, wrecking every AI driven car at once. Ala the Y2K bug if no one knew it was coming, and so didn't prepare. In the book "Realms of Light" by Lawrence Watt-Evans a character keeps an electricity shut off switch behind human operated doors in the event that AI goes haywire. I hope we always engineer in such mechanisms. Even the AIs should want such mechanisms, the same way we want emergency services to exist. A self-caring AI wouldn't want to be screwed over or destroyed because of a flaw in its programming.

satvikpendem 3y ago

As long as fewer humans and dogs die from autonomous vehicles than by human-driven vehicles, they are a net good. Don't expect perfection when we don't even expect that from humans.

anonymouskimmer 3y ago
At this point we don't know if this is subtractive, additive, or a wash.
- MattGaiser 3y ago
  How many people would normally be killed in the same distance driven by human drivers?
  - anonymouskimmer 3y ago
    It can't just be the same distance, but the same parameters in general.
    - williamcotton 3y ago
      Less than 1 fatality per at least 100 million miles driven would cover enough scenarios to be an acceptable metric as well as an improvement over human drivers.
  - vba616 3y ago
    I think I remember that a ballpark figure (meaning zero significant figures) for how many miles a human typically drives between fatalities is 100 million, or roughly 100 lifetimes' worth. I also knew a bus driver who had a plaque for going 1 million miles without an accident.
- radq 3y ago
  As someone that rides a bicycle around human-driven vehicles, I will take my chances with autonomous cars. The autonomous car isn't going to drive drunk, use a phone, or drive erratically because the driver is feeling frustrated.

meghan_rain 3y ago

Around a dozen deaths. Basically what would qualify as a mass shooting instead of a mere gun murder.

benatkin 3y ago

If you're thinking of the story I'm thinking of, that wasn't a huge revelation. Someone was supposed to be to take over for the self driving car but didn't.

anonymouskimmer 3y ago
The one with the homeless person crossing the road with a bicycle at night.
- benatkin 3y ago
  Yeah, that's the one, in Arizona. They paid someone relatively well to simply pay attention and the employee unsurprisingly failed at that. What they should have done is found someone who had demonstrated an ability to be relied upon. I never saw them realize that and make that change, and it seems likely it repeated with the dog.
  - anonymouskimmer 3y ago
    Very good point. > What they should have done is found someone who had demonstrated an ability to be relied upon. Two such people. With maybe a third viewing remotely. And no personal device except a work provided dumbphone for emergencies.

wmf 3y ago

Killing one person is an accident, 100 is a disaster, maybe 10,000 would be a large-scale disaster.