6 ms·

Although it's comforting that this exact situation shouldn't happen again in an Uber autonomous car... there is no mechanism to share that learning with the oth

Although it's comforting that this exact situation shouldn't happen again in an Uber autonomous car... there is no mechanism to share that learning with the other car companies. There seriously fucking needs to be a consortium for exactly this purpose: sharing system failures. Also my problem with this is that a human death is functionally treated as finding edge cases that are missing a unit test, and progressing the testing rate of the code... and that really bothers me somehow. We need to avoid treating deaths as progress in the pursuit of better things

goshx 9y ago
Thinking about different scenarios as unit tests it shouldn’t be hard for them to simulate all sorts of different scenarios and share those tests. Perhaps that would become part of a new standard for safety measures in addition to crash tests with dummies. In fact, I really think this will become the norm in the near future. It might even be out there already in some form.
piinbinary 9y ago
That raises an interesting question: Should there be the equivalent of the FAA for self-driving cars? (Perhaps this could be a function of DoT.)
- 0xffff2 9y ago
  Although the FAA is of course involved, it's generally the NTSB that is the lead investigator in airplane crashes. The NTSB already has jurisdiction to investigate highway accidents. There's also NHTSA (which is indeed part of the DoT). It looks to me like we don't need any new agency at all, just a very small expansion of NHTSA's mandate to specifically address the "self-driving" part of self-driving cars.
emodendroket 9y ago
Well, for one thing, I think I have misgivings in part because Uber hasn't really demonstrated an attitude that makes me think they'll be very careful to reduce accidents. (Also, the extent to which they've bet the farm on autonomous driving presents a lot of temptation to cut corners)
- adrianmonk 9y ago
  Circumstances basically force them into betting the farm. Whether they want SDCs or not, if someone scoops them on SDCs, it's an existential threat to their business.
  - emodendroket 9y ago
    Sure, and also paying the drivers is a big part of the reason they aren't making money yet. Nevertheless, I don't think it changes the fact that there is a big temptation for someone to start cutting corners if it's not moving along fast enough.
    - adrianmonk 9y ago
      Oh yeah, I'm not disagreeing with you. I think it actually makes the temptation much worse. They've built this business with billions in revenue, and they will probably lose it all if this SDC doesn't succeed. What's more, a lot of people seem to think Waymo's tech is further along. So not only does this project have to succeed, it is also the underdog. So no wonder they're aggressive.
daenz 9y ago
>There seriously fucking needs to be a consortium for exactly this purpose: sharing system failures. This implies they're using the same systems and underlying models. If one model hit a pedestrian because of a weakness in training data plus a sub-optimal model hyperparameter, and therefore was classified a pedestrian in that specific pose as some trash on the street, how do you share that conclusion with other companies models?
- icc97 9y ago
  I guess it depends on the data available, but I don't think the hyperparameters are what's needed. You just need to see what the car sensed and how it responded. Then the other car companies can try to replicate the similar circumstances and see what their models would do.
  - daenz 9y ago
    I understand, but the error may be in how it interpreted what it sensed. This is callous language, but if the models interpreted the pedestrian as trash on the street, then how it responded (driving over it) is not inappropriate.
    - ctrl-j 9y ago
      > I understand, but the error may be in how it interpreted what it sensed. It may, it also might not. If sharing data fails, further methods would be needed - but it's a good start for figuring out what data should be recorded for comparison. If the data is entirely incompatible, then we should have regulation to require companies to at minimum transcribe the data into a consumable format after an event such as this. > ... if the models interpreted the pedestrian as trash on the street, then how it responded (driving over it) is not inappropriate. If the models saw anything at all, they should not have driven over it. Even human drivers are taught that driving over road debris is dangerous. At minimum it puts the car/occupants at risk - in extreme cases, the driver may not recognize what it is they are driving over. If this isn't a case where the car was physically unable to stop - it's more likely the telemetry didn't identify the person as an obstacle to avoid at all.
zipwitch 9y ago
It makes me uncomfortable too. I think it's because it's a real world trolley problem. We all make similar decisions all of our lives, but nearly always at some remove. (With rare exceptions in fields like medicine and military operations.) But autonomous vehicles are a very visceral and direct implementation. The difference betweeen the trolley problem and in autonomous vehicles is in time delay and the amount of effort and skill required in execution. Plus, we're taking what is pretty clearly a moral decision and putting it into a system that doesn't do moral decision-making.
pj_mukh 9y ago
Anyone know how this is handled between Boeing and Airbus? Can we mandate the same mechanism between Uber and Waymo?
mindslight 9y ago
> We need to avoid treating deaths as progress in the pursuit of better things Au contraire. Go read building codes some time. There's a saying that they're "written in blood" - every bit, no matter how obvious or arbitrary seeming, was earned through some real-world failure. The death itself isn't progress, of course. But we owe it to the person to who died to learn from what happened.
- bigiain 9y ago
  You seriously think Uber wouldn't try to claim "commercial in confidence" or "trade secrets" rights to all the data from every single death?
  - mindslight 9y ago
    Clearly I think Uber is a benevolent entity that has all of our best interests at heart. Also, I eat babies. Or, you know, don't jump on comments for their not explicitly addressing the hobby horse you're riding. Frankly I just wanted to express a better engineering context around the loss of life without getting into the political bullshit for once.
- pooloo1 9y ago
  Federal Aviation Administration's Joint Order 7110.65 is exactly what you are talking about. It is the manual that Air Traffic Controllers live by. To include situations that involve diverging aircraft or taxiing instructions and how to handle said situations. The entire manual was practically written from real-world experience.
- daxelrod 9y ago
  You might enjoy _The History of Fire Escapes: How to Fail_, a talk Tanya Reilly gave at a DevOps Days conference earlier this year. https://www.youtube.com/watch?v=02KEKtc-5Dc https://www.youtube.com/watch?v=02KEKtc-5Dc
- atopuzov 9y ago
  This reminds me of the story behind the iron ring and the collapse of the second narrows bridge in Vancouver, BC. http://www2.ensc.sfu.ca/undergrad/euss/enscquire/Vol10No2/page9_1.html http://www2.ensc.sfu.ca/undergrad/euss/enscquire/Vol10No2/pa...
ekianjo 9y ago
> We need to avoid treating deaths as progress in the pursuit of better things Then by all means lets stay at home and avoid putting humans in rockets ever again because if you think space exploration will be done without deaths you are in for a surprise.
- knuththetruth 9y ago
  Notoriously criminal company killing pedestrians with robo-cars to make money for themselves != Space exploration by volunteer scientists and service members
thfaway 9y ago
>Although it's comforting that this exact situation shouldn't happen again in an Uber autonomous car. Let me ask something. Did it take a human life to show that "a pedestrian crossing unexpectedly" is a probably thing to happen while driving? I mean, seriously? Maybe they have trained the cars for this, but this particular incidence was just different enough, which means that your initial implication will not hold ie "Although it's comforting that this exact situation shouldn't happen again in an Uber autonomous car" given that no two situations are exactly the same, and it won't apply even for similar situations...

goshx 9y ago

Thinking about different scenarios as unit tests it shouldn’t be hard for them to simulate all sorts of different scenarios and share those tests. Perhaps that would become part of a new standard for safety measures in addition to crash tests with dummies. In fact, I really think this will become the norm in the near future. It might even be out there already in some form.

piinbinary 9y ago

That raises an interesting question: Should there be the equivalent of the FAA for self-driving cars? (Perhaps this could be a function of DoT.)

0xffff2 9y ago
Although the FAA is of course involved, it's generally the NTSB that is the lead investigator in airplane crashes. The NTSB already has jurisdiction to investigate highway accidents. There's also NHTSA (which is indeed part of the DoT). It looks to me like we don't need any new agency at all, just a very small expansion of NHTSA's mandate to specifically address the "self-driving" part of self-driving cars.

emodendroket 9y ago

Well, for one thing, I think I have misgivings in part because Uber hasn't really demonstrated an attitude that makes me think they'll be very careful to reduce accidents. (Also, the extent to which they've bet the farm on autonomous driving presents a lot of temptation to cut corners)

adrianmonk 9y ago
Circumstances basically force them into betting the farm. Whether they want SDCs or not, if someone scoops them on SDCs, it's an existential threat to their business.
- emodendroket 9y ago
  Sure, and also paying the drivers is a big part of the reason they aren't making money yet. Nevertheless, I don't think it changes the fact that there is a big temptation for someone to start cutting corners if it's not moving along fast enough.
  - adrianmonk 9y ago
    Oh yeah, I'm not disagreeing with you. I think it actually makes the temptation much worse. They've built this business with billions in revenue, and they will probably lose it all if this SDC doesn't succeed. What's more, a lot of people seem to think Waymo's tech is further along. So not only does this project have to succeed, it is also the underdog. So no wonder they're aggressive.

daenz 9y ago

>There seriously fucking needs to be a consortium for exactly this purpose: sharing system failures. This implies they're using the same systems and underlying models. If one model hit a pedestrian because of a weakness in training data plus a sub-optimal model hyperparameter, and therefore was classified a pedestrian in that specific pose as some trash on the street, how do you share that conclusion with other companies models?

icc97 9y ago
I guess it depends on the data available, but I don't think the hyperparameters are what's needed. You just need to see what the car sensed and how it responded. Then the other car companies can try to replicate the similar circumstances and see what their models would do.
- daenz 9y ago
  I understand, but the error may be in how it interpreted what it sensed. This is callous language, but if the models interpreted the pedestrian as trash on the street, then how it responded (driving over it) is not inappropriate.
  - ctrl-j 9y ago
    > I understand, but the error may be in how it interpreted what it sensed. It may, it also might not. If sharing data fails, further methods would be needed - but it's a good start for figuring out what data should be recorded for comparison. If the data is entirely incompatible, then we should have regulation to require companies to at minimum transcribe the data into a consumable format after an event such as this. > ... if the models interpreted the pedestrian as trash on the street, then how it responded (driving over it) is not inappropriate. If the models saw anything at all, they should not have driven over it. Even human drivers are taught that driving over road debris is dangerous. At minimum it puts the car/occupants at risk - in extreme cases, the driver may not recognize what it is they are driving over. If this isn't a case where the car was physically unable to stop - it's more likely the telemetry didn't identify the person as an obstacle to avoid at all.

zipwitch 9y ago

It makes me uncomfortable too. I think it's because it's a real world trolley problem. We all make similar decisions all of our lives, but nearly always at some remove. (With rare exceptions in fields like medicine and military operations.) But autonomous vehicles are a very visceral and direct implementation. The difference betweeen the trolley problem and in autonomous vehicles is in time delay and the amount of effort and skill required in execution. Plus, we're taking what is pretty clearly a moral decision and putting it into a system that doesn't do moral decision-making.

pj_mukh 9y ago

Anyone know how this is handled between Boeing and Airbus? Can we mandate the same mechanism between Uber and Waymo?

mindslight 9y ago

> We need to avoid treating deaths as progress in the pursuit of better things Au contraire. Go read building codes some time. There's a saying that they're "written in blood" - every bit, no matter how obvious or arbitrary seeming, was earned through some real-world failure. The death itself isn't progress, of course. But we owe it to the person to who died to learn from what happened.

bigiain 9y ago
You seriously think Uber wouldn't try to claim "commercial in confidence" or "trade secrets" rights to all the data from every single death?
- mindslight 9y ago
  Clearly I think Uber is a benevolent entity that has all of our best interests at heart. Also, I eat babies. Or, you know, don't jump on comments for their not explicitly addressing the hobby horse you're riding. Frankly I just wanted to express a better engineering context around the loss of life without getting into the political bullshit for once.
pooloo1 9y ago
Federal Aviation Administration's Joint Order 7110.65 is exactly what you are talking about. It is the manual that Air Traffic Controllers live by. To include situations that involve diverging aircraft or taxiing instructions and how to handle said situations. The entire manual was practically written from real-world experience.
daxelrod 9y ago
You might enjoy _The History of Fire Escapes: How to Fail_, a talk Tanya Reilly gave at a DevOps Days conference earlier this year. https://www.youtube.com/watch?v=02KEKtc-5Dc https://www.youtube.com/watch?v=02KEKtc-5Dc
atopuzov 9y ago
This reminds me of the story behind the iron ring and the collapse of the second narrows bridge in Vancouver, BC. http://www2.ensc.sfu.ca/undergrad/euss/enscquire/Vol10No2/page9_1.html http://www2.ensc.sfu.ca/undergrad/euss/enscquire/Vol10No2/pa...

ekianjo 9y ago

> We need to avoid treating deaths as progress in the pursuit of better things Then by all means lets stay at home and avoid putting humans in rockets ever again because if you think space exploration will be done without deaths you are in for a surprise.

knuththetruth 9y ago
Notoriously criminal company killing pedestrians with robo-cars to make money for themselves != Space exploration by volunteer scientists and service members

thfaway 9y ago

>Although it's comforting that this exact situation shouldn't happen again in an Uber autonomous car. Let me ask something. Did it take a human life to show that "a pedestrian crossing unexpectedly" is a probably thing to happen while driving? I mean, seriously? Maybe they have trained the cars for this, but this particular incidence was just different enough, which means that your initial implication will not hold ie "Although it's comforting that this exact situation shouldn't happen again in an Uber autonomous car" given that no two situations are exactly the same, and it won't apply even for similar situations...