15 ms·

One aspect that comes from this is that now car crashes can be treated more like aircraft crashes. Each self-driving car now has a black box in it with a ton o

by icc97 9y ago

One aspect that comes from this is that now car crashes can be treated more like aircraft crashes. Each self-driving car now has a black box in it with a ton of telemetary. So it's not just "don't drink and drive", knowing that they'll probably reoffend soon anyway. Every crash and especially fatality can be thoroughly investigated and should be prevented from ever happening again. Hopefully there's enough data in the investigation so that Tesla / Waymo and all other car companies can include the circumstances of the failure in their tests.

Maxious 9y ago
NTSB are investigating this incident including electronic recorders https://www.ntsb.gov/investigations/Pages/HWY18FH010.aspx https://www.ntsb.gov/investigations/Pages/HWY18FH010.aspx and they have another investigation from 2017 still to report https://www.ntsb.gov/investigations/Pages/HWY18FH001.aspx https://www.ntsb.gov/investigations/Pages/HWY18FH001.aspx
fudged71 9y ago
Although it's comforting that this exact situation shouldn't happen again in an Uber autonomous car... there is no mechanism to share that learning with the other car companies. There seriously fucking needs to be a consortium for exactly this purpose: sharing system failures. Also my problem with this is that a human death is functionally treated as finding edge cases that are missing a unit test, and progressing the testing rate of the code... and that really bothers me somehow. We need to avoid treating deaths as progress in the pursuit of better things
- goshx 9y ago
  Thinking about different scenarios as unit tests it shouldn’t be hard for them to simulate all sorts of different scenarios and share those tests. Perhaps that would become part of a new standard for safety measures in addition to crash tests with dummies. In fact, I really think this will become the norm in the near future. It might even be out there already in some form.
- piinbinary 9y ago
  That raises an interesting question: Should there be the equivalent of the FAA for self-driving cars? (Perhaps this could be a function of DoT.)
  - 0xffff2 9y ago
    Although the FAA is of course involved, it's generally the NTSB that is the lead investigator in airplane crashes. The NTSB already has jurisdiction to investigate highway accidents. There's also NHTSA (which is indeed part of the DoT). It looks to me like we don't need any new agency at all, just a very small expansion of NHTSA's mandate to specifically address the "self-driving" part of self-driving cars.
- emodendroket 9y ago
  Well, for one thing, I think I have misgivings in part because Uber hasn't really demonstrated an attitude that makes me think they'll be very careful to reduce accidents. (Also, the extent to which they've bet the farm on autonomous driving presents a lot of temptation to cut corners)
  - adrianmonk 9y ago
    Circumstances basically force them into betting the farm. Whether they want SDCs or not, if someone scoops them on SDCs, it's an existential threat to their business.
    - emodendroket 9y ago
      Sure, and also paying the drivers is a big part of the reason they aren't making money yet. Nevertheless, I don't think it changes the fact that there is a big temptation for someone to start cutting corners if it's not moving along fast enough.
      - adrianmonk 9y ago
        Oh yeah, I'm not disagreeing with you. I think it actually makes the temptation much worse. They've built this business with billions in revenue, and they will probably lose it all if this SDC doesn't succeed. What's more, a lot of people seem to think Waymo's tech is further along. So not only does this project have to succeed, it is also the underdog. So no wonder they're aggressive.
- daenz 9y ago
  >There seriously fucking needs to be a consortium for exactly this purpose: sharing system failures. This implies they're using the same systems and underlying models. If one model hit a pedestrian because of a weakness in training data plus a sub-optimal model hyperparameter, and therefore was classified a pedestrian in that specific pose as some trash on the street, how do you share that conclusion with other companies models?
  - icc97 9y ago
    I guess it depends on the data available, but I don't think the hyperparameters are what's needed. You just need to see what the car sensed and how it responded. Then the other car companies can try to replicate the similar circumstances and see what their models would do.
    - daenz 9y ago
      I understand, but the error may be in how it interpreted what it sensed. This is callous language, but if the models interpreted the pedestrian as trash on the street, then how it responded (driving over it) is not inappropriate.
      - ctrl-j 9y ago
        > I understand, but the error may be in how it interpreted what it sensed. It may, it also might not. If sharing data fails, further methods would be needed - but it's a good start for figuring out what data should be recorded for comparison. If the data is entirely incompatible, then we should have regulation to require companies to at minimum transcribe the data into a consumable format after an event such as this. > ... if the models interpreted the pedestrian as trash on the street, then how it responded (driving over it) is not inappropriate. If the models saw anything at all, they should not have driven over it. Even human drivers are taught that driving over road debris is dangerous. At minimum it puts the car/occupants at risk - in extreme cases, the driver may not recognize what it is they are driving over. If this isn't a case where the car was physically unable to stop - it's more likely the telemetry didn't identify the person as an obstacle to avoid at all.
- zipwitch 9y ago
  It makes me uncomfortable too. I think it's because it's a real world trolley problem. We all make similar decisions all of our lives, but nearly always at some remove. (With rare exceptions in fields like medicine and military operations.) But autonomous vehicles are a very visceral and direct implementation. The difference betweeen the trolley problem and in autonomous vehicles is in time delay and the amount of effort and skill required in execution. Plus, we're taking what is pretty clearly a moral decision and putting it into a system that doesn't do moral decision-making.
- pj_mukh 9y ago
  Anyone know how this is handled between Boeing and Airbus? Can we mandate the same mechanism between Uber and Waymo?
- mindslight 9y ago
  > We need to avoid treating deaths as progress in the pursuit of better things Au contraire. Go read building codes some time. There's a saying that they're "written in blood" - every bit, no matter how obvious or arbitrary seeming, was earned through some real-world failure. The death itself isn't progress, of course. But we owe it to the person to who died to learn from what happened.
  - bigiain 9y ago
    You seriously think Uber wouldn't try to claim "commercial in confidence" or "trade secrets" rights to all the data from every single death?
    - mindslight 9y ago
      Clearly I think Uber is a benevolent entity that has all of our best interests at heart. Also, I eat babies. Or, you know, don't jump on comments for their not explicitly addressing the hobby horse you're riding. Frankly I just wanted to express a better engineering context around the loss of life without getting into the political bullshit for once.
  - pooloo1 9y ago
    Federal Aviation Administration's Joint Order 7110.65 is exactly what you are talking about. It is the manual that Air Traffic Controllers live by. To include situations that involve diverging aircraft or taxiing instructions and how to handle said situations. The entire manual was practically written from real-world experience.
  - daxelrod 9y ago
    You might enjoy _The History of Fire Escapes: How to Fail_, a talk Tanya Reilly gave at a DevOps Days conference earlier this year. https://www.youtube.com/watch?v=02KEKtc-5Dc https://www.youtube.com/watch?v=02KEKtc-5Dc
  - atopuzov 9y ago
    This reminds me of the story behind the iron ring and the collapse of the second narrows bridge in Vancouver, BC. http://www2.ensc.sfu.ca/undergrad/euss/enscquire/Vol10No2/page9_1.html http://www2.ensc.sfu.ca/undergrad/euss/enscquire/Vol10No2/pa...
- ekianjo 9y ago
  > We need to avoid treating deaths as progress in the pursuit of better things Then by all means lets stay at home and avoid putting humans in rockets ever again because if you think space exploration will be done without deaths you are in for a surprise.
  - knuththetruth 9y ago
    Notoriously criminal company killing pedestrians with robo-cars to make money for themselves != Space exploration by volunteer scientists and service members
- thfaway 9y ago
  >Although it's comforting that this exact situation shouldn't happen again in an Uber autonomous car. Let me ask something. Did it take a human life to show that "a pedestrian crossing unexpectedly" is a probably thing to happen while driving? I mean, seriously? Maybe they have trained the cars for this, but this particular incidence was just different enough, which means that your initial implication will not hold ie "Although it's comforting that this exact situation shouldn't happen again in an Uber autonomous car" given that no two situations are exactly the same, and it won't apply even for similar situations...
telchar 9y ago
With Uber's reputation, I wouldn't be surprised if they try to write an app to falsify black box telemetry in the event of a crash to put the liability on the victim. Maybe they'll call it "grey box" or "black mirror". Does the NTSB have regulation on how black boxes are allowed to function?
em3rgent0rdr 9y ago
> "Every crash and especially fatality can be thoroughly investigated" Would be better if the code and data was open for public review.
- jryle70 9y ago
  That's knee-jerk reaction that may open a can of worm. Do you need personal details of the victim as well as the driver? Says if the victim had attempted suicide before? at the same crossroad? Or the driver had history of depression? Would that be a violation of their privacy? Would that cause a witch hunt?
  - ric2b 9y ago
    Code and data, not people's personal details.
    - jryle70 9y ago
      Sure. But if the result is inconclusive, do you leave it at that or demand to know more? What exactly does making the data public better than entrusting a competent organization such as NTSB?
      - ric2b 9y ago
        Yes, you leave it at that, why would personal details be relevant? Making it public means more eyes on the data, which can lead to a better understanding of what went wrong.
baron816 9y ago
I hope we see the day when every car crash makes national news and there’s a big NTSB investigation into what happened.
- dotancohen 9y ago
  This is by far the most insightful comment in the entire thread. The real news isn't "Self-Driving Car Kills Pedestrian", the real news is "Self-Driving Car Fatalities are Rare Enough to be World News". I'm one-third a class M planet away from the incident and reading about it.
  - dragonwriter 9y ago
    > The real news isn't "Self-Driving Car Kills Pedestrian", the real news is "Self-Driving Car Fatalities are Rare Enough to be World News". They are rare only because self-driving cars are; I don't think the total driven miles of all self-driving cars are enough that even 1 fatality would be expected if they were human driven; certainly Uber is orders of magnitude below that point taken alone. There are lots of fatalities from human-driven cars, sure, but that's over a truly stupendous number of miles driven.
    - the_cat_kittles 9y ago
      neither of us has the data, but id bet that whatever miles / fatalities metric is, the self driving cars are still in the lead right now
      - dragonwriter 9y ago
        > neither of us has the data, but id bet that whatever miles / fatalities metric is, the self driving cars are still in the lead right now There's different estimates from different sources using slightly different methodology, but they are all in the neighborhood of road fatalities of 1 per 100 million miles traveled. [0] Waymo claims to have reached 5 million miles in February [1], Uber (from other posts in this thread) is around 1 million miles; the whole self-driving industry is nowhere near 100 million, and has one fatality. So it's way worse, as of today, than human driving for fatalities. Of course, it's also way too little data (on the self-driving side) to treat as meaningful in terms of prediciting the current hazard rather than simply measuring the outcomes to date. [0] see, e.g., https://en.m.wikipedia.org/wiki/Transportation_safety_in_the_United_States https://en.m.wikipedia.org/wiki/Transportation_safety_in_the... [1] https://waymo.com/ontheroad/ https://waymo.com/ontheroad/
icc97 9y ago
I'm hijacking my own post, but this is a very relevant MIT lecture on 'Technology, Policy and Vehicle Safety in the Age of AI' [0]. [0]: https://www.youtube.com/watch?v=LDprUza7yT4 https://www.youtube.com/watch?v=LDprUza7yT4
more_corn 9y ago
Every lesson from aviation is earned in blood. This death wasn't necessary though. The Otto/Uber guys have been informed about their car's difficulty sensing and stopping for pedestrians. I know this because I informed them myself when one almost ran me down in a crosswalk in SF. Can't learn anything from your lessons unless you listen. Maybe they can pause and figure out how to actually listen to their reports of unsafe vehicle behavior.
- woofyman 9y ago
  Police are reporting that it never slowed down. Hitting the pedestrian at 40 mph.
  - lokopodium 9y ago
    It's said she's been hit immediately after entering a car lane outside a crosswalk. Quite possibly there was no time for the autopilot to react at all. I hope all video footage for self-driving crashes is mandatorily released.
    - ionforce 9y ago
      Isn't the point of smart vehicles that they have superhuman reaction speed?
      - lokopodium 9y ago
        It is, actuators still have their lag, probably not much different from foot-pedal combo.
        deleted 9y ago
        [deleted]
      - fastball 9y ago
        There are many instances in Asia of people committing suicide by car (jumping in front of a car with no chance for the driver to stop). Not saying this is the case here, but it could be. As others have said, we need to wait until we know more before jumping to conclusions.
skookumchuck 9y ago
> and should be prevented from ever happening again. Take a look at the cost of airplanes, even small ones.
BucketSort 9y ago
Black box with a ton of telemetry being piped into black box models though.
a_d 9y ago
https://news.ycombinator.com/item?id=16547215 https://news.ycombinator.com/item?id=16547215 This comment by Animats in the context of self-driving trucks is quite telling. He warns precisely of this danger.

by icc97 9y ago

Maxious 9y ago

NTSB are investigating this incident including electronic recorders https://www.ntsb.gov/investigations/Pages/HWY18FH010.aspx https://www.ntsb.gov/investigations/Pages/HWY18FH010.aspx and they have another investigation from 2017 still to report https://www.ntsb.gov/investigations/Pages/HWY18FH001.aspx https://www.ntsb.gov/investigations/Pages/HWY18FH001.aspx

fudged71 9y ago

Although it's comforting that this exact situation shouldn't happen again in an Uber autonomous car... there is no mechanism to share that learning with the other car companies. There seriously fucking needs to be a consortium for exactly this purpose: sharing system failures. Also my problem with this is that a human death is functionally treated as finding edge cases that are missing a unit test, and progressing the testing rate of the code... and that really bothers me somehow. We need to avoid treating deaths as progress in the pursuit of better things

goshx 9y ago
Thinking about different scenarios as unit tests it shouldn’t be hard for them to simulate all sorts of different scenarios and share those tests. Perhaps that would become part of a new standard for safety measures in addition to crash tests with dummies. In fact, I really think this will become the norm in the near future. It might even be out there already in some form.
piinbinary 9y ago
That raises an interesting question: Should there be the equivalent of the FAA for self-driving cars? (Perhaps this could be a function of DoT.)
- 0xffff2 9y ago
  Although the FAA is of course involved, it's generally the NTSB that is the lead investigator in airplane crashes. The NTSB already has jurisdiction to investigate highway accidents. There's also NHTSA (which is indeed part of the DoT). It looks to me like we don't need any new agency at all, just a very small expansion of NHTSA's mandate to specifically address the "self-driving" part of self-driving cars.
emodendroket 9y ago
Well, for one thing, I think I have misgivings in part because Uber hasn't really demonstrated an attitude that makes me think they'll be very careful to reduce accidents. (Also, the extent to which they've bet the farm on autonomous driving presents a lot of temptation to cut corners)
- adrianmonk 9y ago
  Circumstances basically force them into betting the farm. Whether they want SDCs or not, if someone scoops them on SDCs, it's an existential threat to their business.
  - emodendroket 9y ago
    Sure, and also paying the drivers is a big part of the reason they aren't making money yet. Nevertheless, I don't think it changes the fact that there is a big temptation for someone to start cutting corners if it's not moving along fast enough.
    - adrianmonk 9y ago
      Oh yeah, I'm not disagreeing with you. I think it actually makes the temptation much worse. They've built this business with billions in revenue, and they will probably lose it all if this SDC doesn't succeed. What's more, a lot of people seem to think Waymo's tech is further along. So not only does this project have to succeed, it is also the underdog. So no wonder they're aggressive.
daenz 9y ago
>There seriously fucking needs to be a consortium for exactly this purpose: sharing system failures. This implies they're using the same systems and underlying models. If one model hit a pedestrian because of a weakness in training data plus a sub-optimal model hyperparameter, and therefore was classified a pedestrian in that specific pose as some trash on the street, how do you share that conclusion with other companies models?
- icc97 9y ago
  I guess it depends on the data available, but I don't think the hyperparameters are what's needed. You just need to see what the car sensed and how it responded. Then the other car companies can try to replicate the similar circumstances and see what their models would do.
  - daenz 9y ago
    I understand, but the error may be in how it interpreted what it sensed. This is callous language, but if the models interpreted the pedestrian as trash on the street, then how it responded (driving over it) is not inappropriate.
    - ctrl-j 9y ago
      > I understand, but the error may be in how it interpreted what it sensed. It may, it also might not. If sharing data fails, further methods would be needed - but it's a good start for figuring out what data should be recorded for comparison. If the data is entirely incompatible, then we should have regulation to require companies to at minimum transcribe the data into a consumable format after an event such as this. > ... if the models interpreted the pedestrian as trash on the street, then how it responded (driving over it) is not inappropriate. If the models saw anything at all, they should not have driven over it. Even human drivers are taught that driving over road debris is dangerous. At minimum it puts the car/occupants at risk - in extreme cases, the driver may not recognize what it is they are driving over. If this isn't a case where the car was physically unable to stop - it's more likely the telemetry didn't identify the person as an obstacle to avoid at all.
zipwitch 9y ago
It makes me uncomfortable too. I think it's because it's a real world trolley problem. We all make similar decisions all of our lives, but nearly always at some remove. (With rare exceptions in fields like medicine and military operations.) But autonomous vehicles are a very visceral and direct implementation. The difference betweeen the trolley problem and in autonomous vehicles is in time delay and the amount of effort and skill required in execution. Plus, we're taking what is pretty clearly a moral decision and putting it into a system that doesn't do moral decision-making.
pj_mukh 9y ago
Anyone know how this is handled between Boeing and Airbus? Can we mandate the same mechanism between Uber and Waymo?
mindslight 9y ago
> We need to avoid treating deaths as progress in the pursuit of better things Au contraire. Go read building codes some time. There's a saying that they're "written in blood" - every bit, no matter how obvious or arbitrary seeming, was earned through some real-world failure. The death itself isn't progress, of course. But we owe it to the person to who died to learn from what happened.
- bigiain 9y ago
  You seriously think Uber wouldn't try to claim "commercial in confidence" or "trade secrets" rights to all the data from every single death?
  - mindslight 9y ago
    Clearly I think Uber is a benevolent entity that has all of our best interests at heart. Also, I eat babies. Or, you know, don't jump on comments for their not explicitly addressing the hobby horse you're riding. Frankly I just wanted to express a better engineering context around the loss of life without getting into the political bullshit for once.
- pooloo1 9y ago
  Federal Aviation Administration's Joint Order 7110.65 is exactly what you are talking about. It is the manual that Air Traffic Controllers live by. To include situations that involve diverging aircraft or taxiing instructions and how to handle said situations. The entire manual was practically written from real-world experience.
- daxelrod 9y ago
  You might enjoy _The History of Fire Escapes: How to Fail_, a talk Tanya Reilly gave at a DevOps Days conference earlier this year. https://www.youtube.com/watch?v=02KEKtc-5Dc https://www.youtube.com/watch?v=02KEKtc-5Dc
- atopuzov 9y ago
  This reminds me of the story behind the iron ring and the collapse of the second narrows bridge in Vancouver, BC. http://www2.ensc.sfu.ca/undergrad/euss/enscquire/Vol10No2/page9_1.html http://www2.ensc.sfu.ca/undergrad/euss/enscquire/Vol10No2/pa...
ekianjo 9y ago
> We need to avoid treating deaths as progress in the pursuit of better things Then by all means lets stay at home and avoid putting humans in rockets ever again because if you think space exploration will be done without deaths you are in for a surprise.
- knuththetruth 9y ago
  Notoriously criminal company killing pedestrians with robo-cars to make money for themselves != Space exploration by volunteer scientists and service members
thfaway 9y ago
>Although it's comforting that this exact situation shouldn't happen again in an Uber autonomous car. Let me ask something. Did it take a human life to show that "a pedestrian crossing unexpectedly" is a probably thing to happen while driving? I mean, seriously? Maybe they have trained the cars for this, but this particular incidence was just different enough, which means that your initial implication will not hold ie "Although it's comforting that this exact situation shouldn't happen again in an Uber autonomous car" given that no two situations are exactly the same, and it won't apply even for similar situations...

telchar 9y ago

With Uber's reputation, I wouldn't be surprised if they try to write an app to falsify black box telemetry in the event of a crash to put the liability on the victim. Maybe they'll call it "grey box" or "black mirror". Does the NTSB have regulation on how black boxes are allowed to function?

em3rgent0rdr 9y ago

> "Every crash and especially fatality can be thoroughly investigated" Would be better if the code and data was open for public review.

jryle70 9y ago
That's knee-jerk reaction that may open a can of worm. Do you need personal details of the victim as well as the driver? Says if the victim had attempted suicide before? at the same crossroad? Or the driver had history of depression? Would that be a violation of their privacy? Would that cause a witch hunt?
- ric2b 9y ago
  Code and data, not people's personal details.
  - jryle70 9y ago
    Sure. But if the result is inconclusive, do you leave it at that or demand to know more? What exactly does making the data public better than entrusting a competent organization such as NTSB?
    - ric2b 9y ago
      Yes, you leave it at that, why would personal details be relevant? Making it public means more eyes on the data, which can lead to a better understanding of what went wrong.

baron816 9y ago

I hope we see the day when every car crash makes national news and there’s a big NTSB investigation into what happened.

dotancohen 9y ago
This is by far the most insightful comment in the entire thread. The real news isn't "Self-Driving Car Kills Pedestrian", the real news is "Self-Driving Car Fatalities are Rare Enough to be World News". I'm one-third a class M planet away from the incident and reading about it.
- dragonwriter 9y ago
  > The real news isn't "Self-Driving Car Kills Pedestrian", the real news is "Self-Driving Car Fatalities are Rare Enough to be World News". They are rare only because self-driving cars are; I don't think the total driven miles of all self-driving cars are enough that even 1 fatality would be expected if they were human driven; certainly Uber is orders of magnitude below that point taken alone. There are lots of fatalities from human-driven cars, sure, but that's over a truly stupendous number of miles driven.
  - the_cat_kittles 9y ago
    neither of us has the data, but id bet that whatever miles / fatalities metric is, the self driving cars are still in the lead right now
    - dragonwriter 9y ago
      > neither of us has the data, but id bet that whatever miles / fatalities metric is, the self driving cars are still in the lead right now There's different estimates from different sources using slightly different methodology, but they are all in the neighborhood of road fatalities of 1 per 100 million miles traveled. [0] Waymo claims to have reached 5 million miles in February [1], Uber (from other posts in this thread) is around 1 million miles; the whole self-driving industry is nowhere near 100 million, and has one fatality. So it's way worse, as of today, than human driving for fatalities. Of course, it's also way too little data (on the self-driving side) to treat as meaningful in terms of prediciting the current hazard rather than simply measuring the outcomes to date. [0] see, e.g., https://en.m.wikipedia.org/wiki/Transportation_safety_in_the_United_States https://en.m.wikipedia.org/wiki/Transportation_safety_in_the... [1] https://waymo.com/ontheroad/ https://waymo.com/ontheroad/

icc97 9y ago

I'm hijacking my own post, but this is a very relevant MIT lecture on 'Technology, Policy and Vehicle Safety in the Age of AI' [0]. [0]: https://www.youtube.com/watch?v=LDprUza7yT4 https://www.youtube.com/watch?v=LDprUza7yT4

more_corn 9y ago

Every lesson from aviation is earned in blood. This death wasn't necessary though. The Otto/Uber guys have been informed about their car's difficulty sensing and stopping for pedestrians. I know this because I informed them myself when one almost ran me down in a crosswalk in SF. Can't learn anything from your lessons unless you listen. Maybe they can pause and figure out how to actually listen to their reports of unsafe vehicle behavior.

woofyman 9y ago
Police are reporting that it never slowed down. Hitting the pedestrian at 40 mph.
- lokopodium 9y ago
  It's said she's been hit immediately after entering a car lane outside a crosswalk. Quite possibly there was no time for the autopilot to react at all. I hope all video footage for self-driving crashes is mandatorily released.
  - ionforce 9y ago
    Isn't the point of smart vehicles that they have superhuman reaction speed?
    - lokopodium 9y ago
      It is, actuators still have their lag, probably not much different from foot-pedal combo.
      - deleted 9y ago
        [deleted]
    - fastball 9y ago
      There are many instances in Asia of people committing suicide by car (jumping in front of a car with no chance for the driver to stop). Not saying this is the case here, but it could be. As others have said, we need to wait until we know more before jumping to conclusions.

skookumchuck 9y ago

> and should be prevented from ever happening again. Take a look at the cost of airplanes, even small ones.

BucketSort 9y ago

Black box with a ton of telemetry being piped into black box models though.

a_d 9y ago

https://news.ycombinator.com/item?id=16547215 https://news.ycombinator.com/item?id=16547215 This comment by Animats in the context of self-driving trucks is quite telling. He warns precisely of this danger.