20 ms·

Direct link to NTSB preliminary report: https://www.ntsb.gov/investigations/AccidentReports/Reports/HWY18MH010-prelim.pdf https://www.ntsb.gov/investigations/Ac

Direct link to NTSB preliminary report: https://www.ntsb.gov/investigations/AccidentReports/Reports/HWY18MH010-prelim.pdf https://www.ntsb.gov/investigations/AccidentReports/Reports/... According to data obtained from the self-driving system, the system first registered radar and LIDAR observations of the pedestrian about 6 seconds before impact, when the vehicle was traveling at 43 mph. As the vehicle and pedestrian paths converged, the self-driving system software classified the pedestrian as an unknown object, as a vehicle, and then as a bicycle with varying expectations of future travel path. At 1.3 seconds before impact, the self-driving system determined that an emergency braking maneuver was needed to mitigate a collision (see figure 2). 2 According to Uber, emergency braking maneuvers are not enabled while the vehicle is under computer control, to reduce the potential for erratic vehicle behavior. The vehicle operator is relied on to intervene and take action. The system is not designed to alert the operator.

bytematic 8y ago
That makes no sense
- 762236 8y ago
  I suspect that Uber was optimizing for the common case, which is normal traffic conditions, and didn't want their emergency braking accidentally firing during normal driving causing rear endings. So the question becomes why couldn't they get emergency braking solved before driving on the road? Maybe that requires collecting good data first, for training the system?
  - mannykannot 8y ago
    You don't make things safe by optimizing for the common case, and "it can't be safe until after we have tested it on the road" is not a valid reason to allow testing of an unsafe vehicle on public roads.
  - namibj 8y ago
    Considering that 1) you are supposed to keep sufficient distance to break even if the vehicle/generic object in front of you suddenly freezes in space, e.g. a massive wall flush with the rear end of the vehicle in front of you suddenly appears, and 2) the only things that could conceivably rear end an emergency breaking Uber/Volvo would be bikes and other >=4 wheel vehicles (cars/trucks/etc.), which either drive carefully anyway ((non-)motorized bikes), or have a low probability of human damage (cars/trucks), false positives should be preferred to false negatives by somewhere between 5:1 and 1000:1. The latter only if the following vehicles are civilized enough to hold the distance. The car could figure this in, and computer probabilities for what damage an emergency maneuver would cause, which means that it'd break for a stray cat if it's otherwise alone on the road and the surface is dry. But it won't break for a wolf if it's tailgated by a truck, but potentially even accelerate, to make up for the loss of momentum (but light the rear break lights, to stop as soon as it registers the truck to slow down).
    - RyanCavanaugh 8y ago
      b r a k e
  - empath75 8y ago
    > didn't want their emergency braking accidentally firing during normal driving causing rear endings. In no case would sudden braking be the cause of a rear-end collision. It's always the fault of the driver behind.
    - EpicEng 8y ago
      True, and I'm with you, though I'd bet every dollar I have that, if people routinely randomly slammed on their brakes we'd have a whole lot more rear end collisions. We don't expect people to do that, and when it does happen it's rare.
  - skrause 8y ago
    > I suspect that Uber was optimizing for the common case, which is normal traffic conditions That's like creating HTML forms that only work with the common use case and crash spectacularly on unexpected input. Except that this time it's fatal. That's not the kind of software quality I want to see on roads.
  - djsumdog 8y ago
    Why not license the software system from Volvo, which obviously works (people drive Volvos with this turned on every day without erratic breaking), instead of disabling theirs because it was apparently broken? This is beyond incompetence. There is a different level of software engineering when making a website vs making a pacemaker, rocket or flight avionics. You need the quality control of NASA, SpaceX or Boeing, not that of .. whoever they have running their self driving division.
    - criddell 8y ago
      I have a Subaru with EyeSight and it does strange things sometimes. For example, if I happen to be in the left (passing) lane going around a leftward curve and a car in the right lane is stopping or slowing to turn right, the Subaru will hit the brakes because due to the curve of the road, the right lane car is straight ahead. It's scared me a few times. The other thing about the system that sucks is that it's all optical (AFAIK) so when visibility is poor, it shuts off. They need to add more sensors because those are the conditions I would most like an extra set of eyes.
    - wiz21c 8y ago
      Goddmn right : if the things brakes erratically then, come one, it's not ready to brake !!
      - FartyMcFarter 8y ago
        s/not ready to brake/not ready to drive/
  - EpicEng 8y ago
    > I suspect that Uber was optimizing for the common case Yeah that's not how you design safety critical software. This isn't some web service. Either you're wrong (let's hope) or Uber is completely negligent. Source: I write safety critical code for a living.
RyanCavanaugh 8y ago
That quote is the crux of it when you pair it with this other section: "In addition, the operator is responsible for monitoring diagnostic messages that appear on an interface in the center stack of the vehicle dash and tagging events of interest for subsequent review." So you have a "driver" who has to be monitoring a diagnostic console, AND has to be separately watching for non-alerted emergency events to avoid a fatal crash? Why not hire two people? Good god.
- JorgeGT 8y ago
  Move fast and disable the brakes. They in fact began testing with two people per car, but then decided to go with just one. An Uber spokeswoman stated [1]: > We decided to make this transition [from two to one] because after testing, we felt we could accomplish the task of the second person—annotating each intervention with information about what was happening around the car—by looking at our logs after the vehicle had returned to base, rather than in real time. However, this seems to contradict the NTSB report which indicates that it still was the driver's responsibility to perform this event tagging task, which necessarily implies taking your eyes off the road. [1] https://www.citylab.com/transportation/2018/03/former-uber-backup-driver-we-saw-this-coming/556427/ https://www.citylab.com/transportation/2018/03/former-uber-b...
  - majewsky 8y ago
    > which necessarily implies taking your eyes off the road Don't we have speech-to-text for this sort of thing?
- niftich 8y ago
  Initially, they did [1]. "Uber moved from two employees in every car to one. The paired employees had been splitting duties — one ready to take over if the autonomous system failed, and another to keep an eye on what the computers were detecting. The second person was responsible for keeping track of system performance as well as labeling data on a laptop computer. Mr. Kallman, the Uber spokesman, said the second person was in the car for purely data related tasks, not safety." [1] https://www.nytimes.com/2018/03/23/technology/uber-self-driving-cars-arizona.html https://www.nytimes.com/2018/03/23/technology/uber-self-driv...
Voloskaya 8y ago
tl;dr: Safety wasn't a concern for Uber, and they knowlingly put a vehicle with a broken self-driving system on the road. I hope this won't stay unpunished (both at corporate and personal level) if confirmed.
- sschueller 8y ago
  Travis although no longer at Uber should also be held responsible as he was the driving force behind the policies and culture of 'breaking the law' to get ahead.
  - reacharavindh 8y ago
    I hope at least one human being in touch with him tells Travis that he is directly responsible for the death of a fellow human being. The person that died deserves at least that few seconds of remorse.
beamatronic 8y ago
That is absolute fucking insanity.
emodendroket 8y ago
Incredible. What is this software doing on real roads?
- djsumdog 8y ago
  It shows that Uber is the exact same company it's been for years. Nothing has changed. For something this safety critical, you want the software engineering quality of Boeing, NASA, etc. This type of mistake is pretty inexcusable.
  - iClaudiusX 8y ago
    Even more egregious that the governor, Doug Ducey, was happy to issue executive orders to waive any safety oversight to allow Uber to put the public at risk in the first place.[0] https://www.nbcnews.com/tech/innovation/emails-show-arizona-governor-s-cozy-relationship-uber-n860996 https://www.nbcnews.com/tech/innovation/emails-show-arizona-...
  - deleted 8y ago
    [deleted]
- masklinn 8y ago
  AZ decided regulations were unnecessary overhead.
  - agumonkey 8y ago
    Or maybe people hand waved around software review saying "its fiiiiine" leveraging trust or .. $trust$
- PeterisP 8y ago
  Move fast and break things (and people).
dsfyu404ed 8y ago
>At 1.3 seconds before impact, the self-driving system determined that an emergency braking maneuver was needed to mitigate a collision (see figure 2). >The system is not designed to alert the operator. >The vehicle operator intervened less than a second before impact by engaging the steering wheel. >She had been monitoring the self-driving system interface It seems like this was really aggravated by bad UX. Had the system alerted the user and the user had a big red "take whatever emergency action you think is best or stop ASAP if you don't know what to do" button to mash this could would have had a much better chance of being avoided. Things coming onto the road unexpectedly isn't exactly an edge case when it comes to crash causing situations. I don't see why they wouldn't at least alert the user if the system detects a possible collision with an object coming from the side and the object is classified as one of a certain type (pedestrian, bike other vehicle, etc, no need to alert for things classified as plastic bags). I don't see why they disabled the Volvo system. If they were setting up mannequins in a parking lot and teaching the AI to slalom around them I can see why that might be useful but I don't see why they would want to override the Volvo system when on the road. At the very least the cases where the systems disagree are useful for analysis.
- bluGill 8y ago
  Humans cannot react fast enough. There was just 1.3 seconds allowed. Drivers ed teaches 2 second following distance for a reason: it takes you most of that time to realize there is a problem and get your foot on the brake. In the best case a human would just hit the "okay to stop" button as the accident happened. Of course cars cannot change speed instantly anyway - it is likely that even if the button was hit in time the accident was still unavoidable at 1.3 seconds. The car should have been slowing down hard long before it knew what the danger was. (I haven't read the report - it may or may not have been possible for the computer to avoid the accident)
  - davrosthedalek 8y ago
    Average reaction time for humans for a visual alarm is 1/4 second. So more than a second left. And that's a second at the current speed. At 60 mph (was that the speed?), that's about 25 meters. With 1 g breaking for 1 s, the car's speed could have been reduced to 27 mph, and the person on the street would have had almost 0.4s more time to jump away, alerted by the sound of the car breaking.
ZuLuuuuuu 8y ago
I understand that emergency maneuver system was disabled so the car did not brake between t minus 1.3 and t. But why didn't it brake from t minus 6 to t minus 1.3? Looks like it detected that the car's and object's paths were converging, so why didn't it brake during that interval?
- perl4ever 8y ago
  I observe that cars and other objects often come very close to each other, so it would seem impossible to simply brake based on "converging paths". It's necessary to know what an object is and how it's going to behave. If you don't, I don't see how you can go anywhere.
  - slavik81 8y ago
    You don't need to stop if you're uncertain, but you should slow down. That makes it easier to stop once you know you have to.
    - w0m 8y ago
      with significant fuzz factor, agreed. If i'm on the sidewalk and take a step toward the road, should it make the car jerk? Probably not, it's a hard call for passenger comfort. From another angle, think of subway tracks - the algo your describing would slow to a crawl as it crosses every station.
      - fabianhjr 8y ago
        > From another angle, think of subway tracks - the algo your describing would slow to a crawl as it crosses every station. As far as trains go, they do slow down when passing trough a track adjacent to a platform. There are some non-platform adjacent tracks the train companies use to avoid slowing down, however they will slow down or even stop if something is going on. Similarly, high speed rail doesn't have level crossings due to safety considerations. Overall trains are very safe and they are _designed_ for safety. It is highly irresponsible and immoral to just wing it with people's life/safety.
jandrese 8y ago
Ok, so the AI is too panicky and will brake for no apparent reason so they have to disable that bit while they work on it. Fine. But why the hell wouldn't you have the thing beep to alert the driver that the AI thinks there is a problem and they need to pay extra attention? In fact it seems like this would be helpful when trying to fine tune the system.
- LoSboccacc 8y ago
  What strikes me odd is that once an unknown is detected on the road, the car should already be alerting the driver and slowing down. That buys you time for the ai classifier to do its thing and isn’t as dangerous as an emergency braking later on, so seems a sensible behavior all around.
  - tomkarlo 8y ago
    I'd guess that "unknown objects" happen all the time - it seems like that's the default until something is classified, so tire scrap or plastic bag would also fall into that category. If the car slowed down every time it saw one it would never get anywhere, it should only slow if the object gets classified as something you can't hit and is clearly in the path of the vehicle. Seems like that decision happened too late here, requiring emergency braking... which was disabled (!).
    - amenod 8y ago
      The missing bit here is "...unknown is detected on the road" - if there is a tire scrap or plastic bag or anything that looks suspicious a normal human driver would slow down and give it extra attention, then try to avoid it anyway. You don't drive over / through an object unless you know that it is not harmful, and even then you try to avoid it so you don't drive over a bag... of nails.
      - EpicEng 8y ago
        It's not that simple. Humans can also predict movement, and that is necessary because cars don't stop instantaneously. So you have a person walking toward the street and your dumb smart car is constantly hitting the brakes. This tech simply isn't there yet and I doubt it's all that close.
exelius 8y ago
I think this will end up with an Uber exec facing manslaughter charges for gross negligence.
- eropple 8y ago
  I hope you're right, but that's only part of the solution. Everybody in that reporting chain should be looking down the barrel of consequences. Proportional to their level of control, but harsh to be sure. Implementors need to have it made very clear to them as well that no, just-following-orders isn't enough. I have fired clients for doing reckless and stupid things orders of magnitude reckless and stupid than what Uber has done here and I would hope that I would walk the hell out were I confronted with "we disabled the brake for a smoother ride and then disabled the alarms because they were too noisy". Do thou likewise, yeah?
  - exelius 8y ago
    I think this case is a bit too hard to prove much more than negligence, but establishing criminal liability would send the right message that you gotta do this right before unleashing it on public streets.
  - jrochkind1 8y ago
    Software engineers definitely got to start taking their ethical responsibilities seriously.
- rmrfrmrf 8y ago
  I don't think so. The police have said that 1) the pedestrian was at least partially at-fault for not crossing at a crosswalk and 2) given the circumstances, the same outcome would have occurred with a human driver.
  - deleted 8y ago
    [deleted]
  - amenod 8y ago
    Do you have a link or quote for 2)? I was under impression that while the video looks dark, it wasn't quite so dark in reality and human driver would have fared better (if they were driving instead of checking console every 5 seconds, that is).
dang 8y ago
Thanks, we've changed the URL to that from https://www.wsj.com/articles/uber-self-driving-car-that-struck-killed-pedestrian-wasnt-set-to-stop-in-an-emergency-1527170145 https://www.wsj.com/articles/uber-self-driving-car-that-stru....
eternalban 8y ago
That somehow made me think of Mongo DB ..
- deleted 8y ago
  [deleted]
jackconnor 8y ago
So, in the case that emergency braking is, nothing is designed to happen and no one is informed. I guess they just hoped really hard that it wouldn't murder anyone?
nkoren 8y ago
I worked on the autonomous pod system at Heathrow airport[1]. We used a very conservative control methodology; essentially the vehicle would remain stopped unless it received a positive "GO" signal from multiple independent sensor and control systems. The loss of any "GO" signal would result in an emergency stop. It was very challenging to get those all of those "GO" indicators reliable enough to prevent false positives and constant emergency-braking. The reason we were ultimately able to do this is because we were operating in a fully-segregated environment of our own design. We could be certain that every other vehicle in the system was something that should be fully under our control, so anything even slightly anomalous should be treated as a hazard situation. There are a lot of limitations to this approach, but I'm confident that it could carry literally billions of passengers without a fatality. It is overwhelmingly safe. Operating in a mixed environment is profoundly different. The control system logic is fully reversed: you must presume that it is safe to proceed unless a "STOP" signal received. And because the interpretation of image & LIDAR data is a rather... fuzzy... process, that "STOP" signal needs to have fairly liberal thresholds, otherwise your vehicle will not move. Uber made a critical mistake in counting on a human-in-the-loop to suddenly take control of the vehicle (note: this is why Type 3 automation is something I'm very dubious about), but it's important to understand that if you want autonomous vehicles to move through mixed-mode environments at the speeds which humans drive, then it is absolutely necessary for them to take a fuzzy, probabilistic approach to safety. This will inevitably result in fatalities -- almost certainly fewer than when humans drive, but plenty of fatalities nonetheless. The design of the overall system is system is inherently unsafe. Do you find this unacceptable? If so, then then ultimately the only way to address this is through changing the design of the streets and/or our rules about how they are be used. These are fundamentally infrastructural issues. Merely swapping out vehicle control systems -- robot vs. human -- will be less revolutionary than many expect. 1: http://www.ultraglobalprt.com/ http://www.ultraglobalprt.com/
- mcguire 8y ago
  I'm with you up until the "almost certainly" part. Is there any actual hard data on that?
- aetherson 8y ago
  It is the case that even very good driverless cars of the future will cause fatalities now and then. Even if they're safer than human drivers. Don't conflate that with Uber's screw-up here. This wasn't a situation where a fatality was unavoidable or where a very safe system had a once-in-a-blue-moon problem. It's one where they just drove around not-very-safe cars.
jacquesm 8y ago
That's not just one error but a whole book of errors, and that last bit combined with the reliance on the operator to take action is criminal. (And if it isn't it should be.) I hope that whoever was responsible for this piece of crap software loses a lot of sleep over it, and that Uber will admit that they have no business building safety critical software. Idiots. For 6 seconds the system had crucial information and failed to relay it, for 1.3 seconds the system knew an accident was going to happen and failed to act on that knowledge. Drunk drivers suck, but this is much worse. This is the equivalent of plowing into a pedestrian with a vehicle while you're in full control of it because you are afraid that your perception of the world is so crappy that you will over-react to such situations often enough that the risk of killing someone you know is there is perceived as the lower one. Not to mention all the errors in terms of process and oversight that allowed this p.o.s. software to be deployed in traffic.
- amartya916 8y ago
  "for 1.3 seconds the system knew an accident was going to happen and failed to act on that knowledge." This is so tragic. Even Volvo's own collision avoidance system would (could?) have mitigated the crash a fair bit. From Volvo's own spec. sheet [1]: "For speeds between 45 and 70 km/h, the collision is mitigated." In this case, the NTSB reports mentions that the car was traveling at 43mph, i.e. 68.8 kmph :(. What bothers me is that these systems are on public roads, without public oversight. Sure, Uber got permission from the local authorities, but getting an independent team of technologists and ethicists to sign off on the basic parameters should have been the bare minimum ... yes, that would take time, but do we really want to give companies, especially ones like Uber with a history of ethical transgressions, the benefit of the doubt? [1] https://tinyurl.com/y9sp2fmu https://tinyurl.com/y9sp2fmu (WARNING: This open/downloads a PDF that I referred to above. Page 5 has the paragraph on pedestrian collision detection specs)
- tomkarlo 8y ago
  Is this necessarily different from a car placed in normal cruise control (automatic throttle, no braking), where the driver is under the obligation to managing braking in an emergency? It seems like the human driver here was still under that obligation, but failed to act. (Possibly because they were distracted, but that's not unique to this situation.)
carapace 8y ago
I actually feel sick... > 1.3 seconds before impact ... emergency braking maneuver was needed ... not enabled ... to reduce the potential for erratic vehicle behavior This wind-up toy killed a person. Transport is a waking nightmare anyway. Every time you get in your car, every mile you drive, you're buying a ticket in a horrifying lottery. If you lose the lottery you reach your destination. If you win... blood, pain, death. Into this we're setting loose these badly-programmed projections of our science-fiction. - - - - A sane "greenfield" transportation network would begin with three separate networks, one each for pedestrians, cyclists, and motor vehicles. (As long as I'm dreaming of sane urban infrastructure, let me sing the praises of C. Alexander's "Pattern Language" et. al., and specifically the "Alternating Fingers of City and Country" pattern!) My mom has dementia and is losing her mind. We don't trust her to take the bus across town anymore, and she hasn't driven in years. If I wanted an auto-auto[1] to take her places safely I could build that today. It would be limited to about three miles an hour with a big ol' smiley sign on the back saying "Go Around Asshole" in nicer language. Obviously, you would restrict it to routes that didn't gum up major roads. It would be approximately an electric scooter wrapped in safety mechanisms and encased in a carbon fiber monocoque hull. I can't recall the name now but there's a way to set up impact dampers so that if the hull is hit most of the kinetic energy is absorbed into flywheels (as opposed to bouncing the occupant around like a rag doll or hitting them with explosive pillows.) This machine would pick its way across the city like "an old man crossing a river in winter." Its maximum speed would at all times be set by the braking distance to any possible obstacle. [1] I maintain that "auto-auto" is the obviously cromulent name for self-driving automobiles, and will henceforth use the term unabashedly.

bytematic 8y ago

That makes no sense

762236 8y ago
I suspect that Uber was optimizing for the common case, which is normal traffic conditions, and didn't want their emergency braking accidentally firing during normal driving causing rear endings. So the question becomes why couldn't they get emergency braking solved before driving on the road? Maybe that requires collecting good data first, for training the system?
- mannykannot 8y ago
  You don't make things safe by optimizing for the common case, and "it can't be safe until after we have tested it on the road" is not a valid reason to allow testing of an unsafe vehicle on public roads.
- namibj 8y ago
  Considering that 1) you are supposed to keep sufficient distance to break even if the vehicle/generic object in front of you suddenly freezes in space, e.g. a massive wall flush with the rear end of the vehicle in front of you suddenly appears, and 2) the only things that could conceivably rear end an emergency breaking Uber/Volvo would be bikes and other >=4 wheel vehicles (cars/trucks/etc.), which either drive carefully anyway ((non-)motorized bikes), or have a low probability of human damage (cars/trucks), false positives should be preferred to false negatives by somewhere between 5:1 and 1000:1. The latter only if the following vehicles are civilized enough to hold the distance. The car could figure this in, and computer probabilities for what damage an emergency maneuver would cause, which means that it'd break for a stray cat if it's otherwise alone on the road and the surface is dry. But it won't break for a wolf if it's tailgated by a truck, but potentially even accelerate, to make up for the loss of momentum (but light the rear break lights, to stop as soon as it registers the truck to slow down).
  - RyanCavanaugh 8y ago
    b r a k e
- empath75 8y ago
  > didn't want their emergency braking accidentally firing during normal driving causing rear endings. In no case would sudden braking be the cause of a rear-end collision. It's always the fault of the driver behind.
  - EpicEng 8y ago
    True, and I'm with you, though I'd bet every dollar I have that, if people routinely randomly slammed on their brakes we'd have a whole lot more rear end collisions. We don't expect people to do that, and when it does happen it's rare.
- skrause 8y ago
  > I suspect that Uber was optimizing for the common case, which is normal traffic conditions That's like creating HTML forms that only work with the common use case and crash spectacularly on unexpected input. Except that this time it's fatal. That's not the kind of software quality I want to see on roads.
- djsumdog 8y ago
  Why not license the software system from Volvo, which obviously works (people drive Volvos with this turned on every day without erratic breaking), instead of disabling theirs because it was apparently broken? This is beyond incompetence. There is a different level of software engineering when making a website vs making a pacemaker, rocket or flight avionics. You need the quality control of NASA, SpaceX or Boeing, not that of .. whoever they have running their self driving division.
  - criddell 8y ago
    I have a Subaru with EyeSight and it does strange things sometimes. For example, if I happen to be in the left (passing) lane going around a leftward curve and a car in the right lane is stopping or slowing to turn right, the Subaru will hit the brakes because due to the curve of the road, the right lane car is straight ahead. It's scared me a few times. The other thing about the system that sucks is that it's all optical (AFAIK) so when visibility is poor, it shuts off. They need to add more sensors because those are the conditions I would most like an extra set of eyes.
  - wiz21c 8y ago
    Goddmn right : if the things brakes erratically then, come one, it's not ready to brake !!
    - FartyMcFarter 8y ago
      s/not ready to brake/not ready to drive/
- EpicEng 8y ago
  > I suspect that Uber was optimizing for the common case Yeah that's not how you design safety critical software. This isn't some web service. Either you're wrong (let's hope) or Uber is completely negligent. Source: I write safety critical code for a living.

RyanCavanaugh 8y ago

That quote is the crux of it when you pair it with this other section: "In addition, the operator is responsible for monitoring diagnostic messages that appear on an interface in the center stack of the vehicle dash and tagging events of interest for subsequent review." So you have a "driver" who has to be monitoring a diagnostic console, AND has to be separately watching for non-alerted emergency events to avoid a fatal crash? Why not hire two people? Good god.

JorgeGT 8y ago
Move fast and disable the brakes. They in fact began testing with two people per car, but then decided to go with just one. An Uber spokeswoman stated [1]: > We decided to make this transition [from two to one] because after testing, we felt we could accomplish the task of the second person—annotating each intervention with information about what was happening around the car—by looking at our logs after the vehicle had returned to base, rather than in real time. However, this seems to contradict the NTSB report which indicates that it still was the driver's responsibility to perform this event tagging task, which necessarily implies taking your eyes off the road. [1] https://www.citylab.com/transportation/2018/03/former-uber-backup-driver-we-saw-this-coming/556427/ https://www.citylab.com/transportation/2018/03/former-uber-b...
- majewsky 8y ago
  > which necessarily implies taking your eyes off the road Don't we have speech-to-text for this sort of thing?
niftich 8y ago
Initially, they did [1]. "Uber moved from two employees in every car to one. The paired employees had been splitting duties — one ready to take over if the autonomous system failed, and another to keep an eye on what the computers were detecting. The second person was responsible for keeping track of system performance as well as labeling data on a laptop computer. Mr. Kallman, the Uber spokesman, said the second person was in the car for purely data related tasks, not safety." [1] https://www.nytimes.com/2018/03/23/technology/uber-self-driving-cars-arizona.html https://www.nytimes.com/2018/03/23/technology/uber-self-driv...

Voloskaya 8y ago

tl;dr: Safety wasn't a concern for Uber, and they knowlingly put a vehicle with a broken self-driving system on the road. I hope this won't stay unpunished (both at corporate and personal level) if confirmed.

sschueller 8y ago
Travis although no longer at Uber should also be held responsible as he was the driving force behind the policies and culture of 'breaking the law' to get ahead.
- reacharavindh 8y ago
  I hope at least one human being in touch with him tells Travis that he is directly responsible for the death of a fellow human being. The person that died deserves at least that few seconds of remorse.

beamatronic 8y ago

That is absolute fucking insanity.

emodendroket 8y ago

Incredible. What is this software doing on real roads?

djsumdog 8y ago
It shows that Uber is the exact same company it's been for years. Nothing has changed. For something this safety critical, you want the software engineering quality of Boeing, NASA, etc. This type of mistake is pretty inexcusable.
- iClaudiusX 8y ago
  Even more egregious that the governor, Doug Ducey, was happy to issue executive orders to waive any safety oversight to allow Uber to put the public at risk in the first place.[0] https://www.nbcnews.com/tech/innovation/emails-show-arizona-governor-s-cozy-relationship-uber-n860996 https://www.nbcnews.com/tech/innovation/emails-show-arizona-...
- deleted 8y ago
  [deleted]
masklinn 8y ago
AZ decided regulations were unnecessary overhead.
- agumonkey 8y ago
  Or maybe people hand waved around software review saying "its fiiiiine" leveraging trust or .. $trust$
PeterisP 8y ago
Move fast and break things (and people).

dsfyu404ed 8y ago

>At 1.3 seconds before impact, the self-driving system determined that an emergency braking maneuver was needed to mitigate a collision (see figure 2). >The system is not designed to alert the operator. >The vehicle operator intervened less than a second before impact by engaging the steering wheel. >She had been monitoring the self-driving system interface It seems like this was really aggravated by bad UX. Had the system alerted the user and the user had a big red "take whatever emergency action you think is best or stop ASAP if you don't know what to do" button to mash this could would have had a much better chance of being avoided. Things coming onto the road unexpectedly isn't exactly an edge case when it comes to crash causing situations. I don't see why they wouldn't at least alert the user if the system detects a possible collision with an object coming from the side and the object is classified as one of a certain type (pedestrian, bike other vehicle, etc, no need to alert for things classified as plastic bags). I don't see why they disabled the Volvo system. If they were setting up mannequins in a parking lot and teaching the AI to slalom around them I can see why that might be useful but I don't see why they would want to override the Volvo system when on the road. At the very least the cases where the systems disagree are useful for analysis.

bluGill 8y ago
Humans cannot react fast enough. There was just 1.3 seconds allowed. Drivers ed teaches 2 second following distance for a reason: it takes you most of that time to realize there is a problem and get your foot on the brake. In the best case a human would just hit the "okay to stop" button as the accident happened. Of course cars cannot change speed instantly anyway - it is likely that even if the button was hit in time the accident was still unavoidable at 1.3 seconds. The car should have been slowing down hard long before it knew what the danger was. (I haven't read the report - it may or may not have been possible for the computer to avoid the accident)
- davrosthedalek 8y ago
  Average reaction time for humans for a visual alarm is 1/4 second. So more than a second left. And that's a second at the current speed. At 60 mph (was that the speed?), that's about 25 meters. With 1 g breaking for 1 s, the car's speed could have been reduced to 27 mph, and the person on the street would have had almost 0.4s more time to jump away, alerted by the sound of the car breaking.

ZuLuuuuuu 8y ago

I understand that emergency maneuver system was disabled so the car did not brake between t minus 1.3 and t. But why didn't it brake from t minus 6 to t minus 1.3? Looks like it detected that the car's and object's paths were converging, so why didn't it brake during that interval?

perl4ever 8y ago
I observe that cars and other objects often come very close to each other, so it would seem impossible to simply brake based on "converging paths". It's necessary to know what an object is and how it's going to behave. If you don't, I don't see how you can go anywhere.
- slavik81 8y ago
  You don't need to stop if you're uncertain, but you should slow down. That makes it easier to stop once you know you have to.
  - w0m 8y ago
    with significant fuzz factor, agreed. If i'm on the sidewalk and take a step toward the road, should it make the car jerk? Probably not, it's a hard call for passenger comfort. From another angle, think of subway tracks - the algo your describing would slow to a crawl as it crosses every station.
    - fabianhjr 8y ago
      > From another angle, think of subway tracks - the algo your describing would slow to a crawl as it crosses every station. As far as trains go, they do slow down when passing trough a track adjacent to a platform. There are some non-platform adjacent tracks the train companies use to avoid slowing down, however they will slow down or even stop if something is going on. Similarly, high speed rail doesn't have level crossings due to safety considerations. Overall trains are very safe and they are _designed_ for safety. It is highly irresponsible and immoral to just wing it with people's life/safety.

jandrese 8y ago

Ok, so the AI is too panicky and will brake for no apparent reason so they have to disable that bit while they work on it. Fine. But why the hell wouldn't you have the thing beep to alert the driver that the AI thinks there is a problem and they need to pay extra attention? In fact it seems like this would be helpful when trying to fine tune the system.

LoSboccacc 8y ago
What strikes me odd is that once an unknown is detected on the road, the car should already be alerting the driver and slowing down. That buys you time for the ai classifier to do its thing and isn’t as dangerous as an emergency braking later on, so seems a sensible behavior all around.
- tomkarlo 8y ago
  I'd guess that "unknown objects" happen all the time - it seems like that's the default until something is classified, so tire scrap or plastic bag would also fall into that category. If the car slowed down every time it saw one it would never get anywhere, it should only slow if the object gets classified as something you can't hit and is clearly in the path of the vehicle. Seems like that decision happened too late here, requiring emergency braking... which was disabled (!).
  - amenod 8y ago
    The missing bit here is "...unknown is detected on the road" - if there is a tire scrap or plastic bag or anything that looks suspicious a normal human driver would slow down and give it extra attention, then try to avoid it anyway. You don't drive over / through an object unless you know that it is not harmful, and even then you try to avoid it so you don't drive over a bag... of nails.
    - EpicEng 8y ago
      It's not that simple. Humans can also predict movement, and that is necessary because cars don't stop instantaneously. So you have a person walking toward the street and your dumb smart car is constantly hitting the brakes. This tech simply isn't there yet and I doubt it's all that close.

exelius 8y ago

I think this will end up with an Uber exec facing manslaughter charges for gross negligence.

eropple 8y ago
I hope you're right, but that's only part of the solution. Everybody in that reporting chain should be looking down the barrel of consequences. Proportional to their level of control, but harsh to be sure. Implementors need to have it made very clear to them as well that no, just-following-orders isn't enough. I have fired clients for doing reckless and stupid things orders of magnitude reckless and stupid than what Uber has done here and I would hope that I would walk the hell out were I confronted with "we disabled the brake for a smoother ride and then disabled the alarms because they were too noisy". Do thou likewise, yeah?
- exelius 8y ago
  I think this case is a bit too hard to prove much more than negligence, but establishing criminal liability would send the right message that you gotta do this right before unleashing it on public streets.
- jrochkind1 8y ago
  Software engineers definitely got to start taking their ethical responsibilities seriously.
rmrfrmrf 8y ago
I don't think so. The police have said that 1) the pedestrian was at least partially at-fault for not crossing at a crosswalk and 2) given the circumstances, the same outcome would have occurred with a human driver.
- deleted 8y ago
  [deleted]
- amenod 8y ago
  Do you have a link or quote for 2)? I was under impression that while the video looks dark, it wasn't quite so dark in reality and human driver would have fared better (if they were driving instead of checking console every 5 seconds, that is).

dang 8y ago

Thanks, we've changed the URL to that from https://www.wsj.com/articles/uber-self-driving-car-that-struck-killed-pedestrian-wasnt-set-to-stop-in-an-emergency-1527170145 https://www.wsj.com/articles/uber-self-driving-car-that-stru....

eternalban 8y ago

That somehow made me think of Mongo DB ..

deleted 8y ago
[deleted]

jackconnor 8y ago

So, in the case that emergency braking is, nothing is designed to happen and no one is informed. I guess they just hoped really hard that it wouldn't murder anyone?

nkoren 8y ago

I worked on the autonomous pod system at Heathrow airport[1]. We used a very conservative control methodology; essentially the vehicle would remain stopped unless it received a positive "GO" signal from multiple independent sensor and control systems. The loss of any "GO" signal would result in an emergency stop. It was very challenging to get those all of those "GO" indicators reliable enough to prevent false positives and constant emergency-braking. The reason we were ultimately able to do this is because we were operating in a fully-segregated environment of our own design. We could be certain that every other vehicle in the system was something that should be fully under our control, so anything even slightly anomalous should be treated as a hazard situation. There are a lot of limitations to this approach, but I'm confident that it could carry literally billions of passengers without a fatality. It is overwhelmingly safe. Operating in a mixed environment is profoundly different. The control system logic is fully reversed: you must presume that it is safe to proceed unless a "STOP" signal received. And because the interpretation of image & LIDAR data is a rather... fuzzy... process, that "STOP" signal needs to have fairly liberal thresholds, otherwise your vehicle will not move. Uber made a critical mistake in counting on a human-in-the-loop to suddenly take control of the vehicle (note: this is why Type 3 automation is something I'm very dubious about), but it's important to understand that if you want autonomous vehicles to move through mixed-mode environments at the speeds which humans drive, then it is absolutely necessary for them to take a fuzzy, probabilistic approach to safety. This will inevitably result in fatalities -- almost certainly fewer than when humans drive, but plenty of fatalities nonetheless. The design of the overall system is system is inherently unsafe. Do you find this unacceptable? If so, then then ultimately the only way to address this is through changing the design of the streets and/or our rules about how they are be used. These are fundamentally infrastructural issues. Merely swapping out vehicle control systems -- robot vs. human -- will be less revolutionary than many expect. 1: http://www.ultraglobalprt.com/ http://www.ultraglobalprt.com/

mcguire 8y ago
I'm with you up until the "almost certainly" part. Is there any actual hard data on that?
aetherson 8y ago
It is the case that even very good driverless cars of the future will cause fatalities now and then. Even if they're safer than human drivers. Don't conflate that with Uber's screw-up here. This wasn't a situation where a fatality was unavoidable or where a very safe system had a once-in-a-blue-moon problem. It's one where they just drove around not-very-safe cars.

jacquesm 8y ago

That's not just one error but a whole book of errors, and that last bit combined with the reliance on the operator to take action is criminal. (And if it isn't it should be.) I hope that whoever was responsible for this piece of crap software loses a lot of sleep over it, and that Uber will admit that they have no business building safety critical software. Idiots. For 6 seconds the system had crucial information and failed to relay it, for 1.3 seconds the system knew an accident was going to happen and failed to act on that knowledge. Drunk drivers suck, but this is much worse. This is the equivalent of plowing into a pedestrian with a vehicle while you're in full control of it because you are afraid that your perception of the world is so crappy that you will over-react to such situations often enough that the risk of killing someone you know is there is perceived as the lower one. Not to mention all the errors in terms of process and oversight that allowed this p.o.s. software to be deployed in traffic.

amartya916 8y ago
"for 1.3 seconds the system knew an accident was going to happen and failed to act on that knowledge." This is so tragic. Even Volvo's own collision avoidance system would (could?) have mitigated the crash a fair bit. From Volvo's own spec. sheet [1]: "For speeds between 45 and 70 km/h, the collision is mitigated." In this case, the NTSB reports mentions that the car was traveling at 43mph, i.e. 68.8 kmph :(. What bothers me is that these systems are on public roads, without public oversight. Sure, Uber got permission from the local authorities, but getting an independent team of technologists and ethicists to sign off on the basic parameters should have been the bare minimum ... yes, that would take time, but do we really want to give companies, especially ones like Uber with a history of ethical transgressions, the benefit of the doubt? [1] https://tinyurl.com/y9sp2fmu https://tinyurl.com/y9sp2fmu (WARNING: This open/downloads a PDF that I referred to above. Page 5 has the paragraph on pedestrian collision detection specs)
tomkarlo 8y ago
Is this necessarily different from a car placed in normal cruise control (automatic throttle, no braking), where the driver is under the obligation to managing braking in an emergency? It seems like the human driver here was still under that obligation, but failed to act. (Possibly because they were distracted, but that's not unique to this situation.)

carapace 8y ago

I actually feel sick... > 1.3 seconds before impact ... emergency braking maneuver was needed ... not enabled ... to reduce the potential for erratic vehicle behavior This wind-up toy killed a person. Transport is a waking nightmare anyway. Every time you get in your car, every mile you drive, you're buying a ticket in a horrifying lottery. If you lose the lottery you reach your destination. If you win... blood, pain, death. Into this we're setting loose these badly-programmed projections of our science-fiction. - - - - A sane "greenfield" transportation network would begin with three separate networks, one each for pedestrians, cyclists, and motor vehicles. (As long as I'm dreaming of sane urban infrastructure, let me sing the praises of C. Alexander's "Pattern Language" et. al., and specifically the "Alternating Fingers of City and Country" pattern!) My mom has dementia and is losing her mind. We don't trust her to take the bus across town anymore, and she hasn't driven in years. If I wanted an auto-auto[1] to take her places safely I could build that today. It would be limited to about three miles an hour with a big ol' smiley sign on the back saying "Go Around Asshole" in nicer language. Obviously, you would restrict it to routes that didn't gum up major roads. It would be approximately an electric scooter wrapped in safety mechanisms and encased in a carbon fiber monocoque hull. I can't recall the name now but there's a way to set up impact dampers so that if the hull is hit most of the kinetic energy is absorbed into flywheels (as opposed to bouncing the occupant around like a rag doll or hitting them with explosive pillows.) This machine would pick its way across the city like "an old man crossing a river in winter." Its maximum speed would at all times be set by the braking distance to any possible obstacle. [1] I maintain that "auto-auto" is the obviously cromulent name for self-driving automobiles, and will henceforth use the term unabashedly.