40 ms·

> 5.6 seconds before impact, it classified her as a vehicle. Then it changed its mind to “other,” then to vehicle again, back to “other,” then to bicycle, then

> 5.6 seconds before impact, it classified her as a vehicle. Then it changed its mind to “other,” then to vehicle again, back to “other,” then to bicycle, then to “other” again, and finally back to bicycle. System can't decide what's happening. > It wasn’t until 1.2 seconds before the impact that the system recognized that the SUV was going to hit Herzberg System is too slow to realize something serious is happening. > That triggered what Uber called “action suppression,” in which the system held off braking for one second A hardcoded 1 second delay during a potential emergency situation. Horrifying. I bet they added it because the system kept randomly thinking something serious was going to happen for a few milliseconds when everything was going fine. If you ever find yourself doing that for a safety critical piece of software, you should stop and reconsider what you are doing. This is a hacky patch over a serious underlying classification issue. You need to fix the underlying problem, not hackily patch over it. How is this not the title of the story? This is so much worse than the "it couldn't see her as a person, only as a bicycle". At least the car would still try to avoid a bicycle, in principle, instead of blindly gliding into it while hoping for the best. > with 0.2 seconds left before impact, the car sounded an audio alarm, and Vasquez took the steering wheel, disengaging the autonomous system. Nearly a full second after striking Herzberg, Vasquez hit the brakes. And then top it off with systemic issues around the backup driver not actually being ready to react.

gambiting 7y ago
>>> 5.6 seconds before impact, it classified her as a vehicle. Then it changed its mind to “other,” then to vehicle again, back to “other,” then to bicycle, then to “other” again, and finally back to bicycle. This is exactly why I keep saying that autonomous vehicles are not 10 or even 20 years away. More like 50-100 years away if that. Same reason as to why famously a group of researchers in the 60s thought that solving computer recognition of objects would take few months at max, and yet in 2019 our best algorithms still think that a sofa covered in a zebra print is actually a zebra with 99% confidence. Had a human been actually paying attention to the road, I can bet they would start breaking/swearving as soon as they saw something, even if they weren't immediately 100% certain that it's a human - a computer won't until it's 99%+ certain, which is too risky assumption considering the state of visual recognition of objects.
SideburnsOfDoom 7y ago
> System can't decide what's happening. > At least the car would still try to avoid a bicycle, in principle, instead of blindly gliding into it while hoping for the best. "Don't know what it is, let's ram it." Never mind not detecting a pedestrian, that in itself is terrifyingly incompetent and negligent.
guenthert 7y ago
As I remember it, the driver got a lot of heat for fumbling with her phone (or 2nd computer?) right before the accident. I don't think however that 1.2s is a bad reaction time for a complex situation. Would it have killed the developers to make the car sound its horn when it gets into this absurd 1s "action suppression" mode?
- Strilanc 7y ago
  > the driver got a lot of heat for fumbling with her phone (or 2nd computer?) right before the accident Based on news stories I found, she was glancing at a television show on her phone [1]. > make the car sound its horn when it gets into this absurd 1s "action suppression" mode? If they added the suppression because there were too many false positives, that would just have resulted in the car honking at apparently arbitrary times. It's just converting the garbage signal from one form into another. It's still too noisy to be reliable. 1: https://www.azcentral.com/story/news/local/tempe/2019/03/17/one-year-after-self-driving-uber-rafaela-vasquez-behind-wheel-crash-death-elaine-herzberg-tempe/1296676002/ https://www.azcentral.com/story/news/local/tempe/2019/03/17/... Vasquez looked down 166 times when the vehicle was in motion, not including times she appeared to be checking gauges [...] In all, they concluded, Vasquez traveled about 3.67 miles total while looking away from the road. [...] starting at 9:16, Vasquez played an episode of “The Voice,” The Blind Auditions, Part 5, on her phone.
  - mcphage 7y ago
    > If they added the suppression because there were too many false positives, that would just have resulted in the car honking at apparently arbitrary times. It's just converting the garbage signal from one form into another. It's still too noisy to be reliable. I love how they went from "our vision system is too unreliable to have warning signals every time it doesn't know what's in front of it" to "okay let's do it anyway but just not have warning signals". Like it didn't make them stop and think "well maybe basing a self-driving car off of this isn't a good idea".
    - EpicEng 7y ago
me_me_me 7y ago
I wonder why pick such vehicle to test this. Why not something with smaller mass to have less momentum on impact. (ie google car).
- yifanl 7y ago
  Actually yeah, can we not rig some kind of bicycle with the same sensors and test self-driving that way? The steering mechanism would have to be modified obviously, but surely steering is a trivial part of the problem compared to actually figuring out where to steer to?
  - rtkwe 7y ago
    Part of path planning will involve vehicle dynamics, breaking, acceleration and steering response, and the envelope of the vehicle. All of those will heavily impact how a car should drive.
    - lovehashbrowns 7y ago
      But that's something the AI can learn fairly easily, isn't it? The difficulty in this case wasn't that the AI had issues figuring out the handling of the SUV, it's that it had issues detecting a pedestrian and a dangerous situation. You can still run into these issues on a bicycle, with a much lesser chance of killing people.
      - dkonofalski 7y ago
        This is totally true but the issue was more with the methodology of the detection rather than the detection itself. Regardless of the type of vehicle, the software wasn't good enough for real-world testing.
  - fgvuyg 7y ago
    cycle dynamics is an unsolved control problem
Strilanc 7y ago
To rant a bit more about this one second delay thing. This reeks of a type of thinking where you are relying on other parts of the system to compensate. You might expect to hear things like "it's okay, the safety driver will catch it". Speaking for myself personally, this type of thinking comes very naturally. I like to come up with little proofs that a problem is handled by one part of a program, and then forget about that problem. But in my experience (which does not involve writing anything safety critical) this strategy kinda sucks at getting things right. Dependencies change, assumptions become invalid, holes in your intuitive proof become apparent, etc, etc, etc, and the thing falls over. If you are designing a safety critical system, something you really want to work, I don't think you should be thinking in the mode where each problem is handled by one part of the system. You need to be thinking in terms of defense in depth. When something goes wrong, many parts of the system should all be able to detect and correct the problem. And then when something bad does come up, and 9 out of 10 of those defensive layers each individually were sufficient to save the day so there was no disaster, you should go figure out what the hell went wrong in the tenth.
- jeromebaek 7y ago
  This. The right way to think is that each component, in parallel, have a chance of succeeding, so chance of total system failure is exponentially small in the number of components. Not: oh if this layer fails, the next one will catch it... which makes the chance of failure as high as the weakest link.
- flowerlad 7y ago
  > This reeks of a type of thinking where you are relying on other parts of the system to compensate. This is what Boeing did with Max. The airframe wasn’t stable in and of itself, and they relied on software to compensate. Terrible idea.
- steelframe 7y ago
  > Dependencies change, assumptions become invalid, holes in your intuitive proof become apparent, etc, etc, etc, and the thing falls over. I apply encryption to storage. I can't tell you how often people try to push back on encrypting storage with stories like, "But we have access controls and auditing in place. And when we have a deprovisioning process for our drives. Encryption is costly and redundant, so why should we do it?" Through the years I can recount several after-the-fact incidents where encryption ended up saving their bacon because of weird and entirely unanticipated events. One notable one was where a hypervisor bug caused memory to persist to an unintended location during suspend/resume, and the only reason customer data wasn't exposed in production was because the storage target was encrypted. In another case the "streams were crossed" when assigning virtual disks to virtual machines. The (older) base disk images weren't encrypted in that case, but because the newer machines were applying encryption in the backend before the blocks were exposed to the guest OS, the "unencrypted" disk content came across as garbage (plaintext was "decrypted," which with the algorithms we were using was equivalent to encrypting), again preserving the confidentiality of the original disk images. The concept of "belt and suspenders" is often lost on people when it comes to safety and security systems.
SilasX 7y ago
>> 5.6 seconds before impact, it classified her as a vehicle. Then it changed its mind to “other,” then to vehicle again, back to “other,” then to bicycle, then to “other” again, and finally back to bicycle. Those still all seem to fall into the category "thing you should avoid hitting", though, right?
- NoodleIncident 7y ago
  The table goes into more detail. Each time the classification changed, the history for that object was essentially deleted; since there's only one data point, the system predicted that it would "continue" to stay stationary, even though the pedestrian was walking at a steady pace.
  - rubicon33 7y ago
    That part, about deleting the history, confuses me. Why delete the history on a classification change? Shouldn't classifications be tiered? In this case, while the system was struggling to PERFECTLY classify the object, it was clearly thinking it was something that should be avoided (oscillating between car, bike, other). In this case, I would expect the system to keep it's motion history. IMO, this could have prevented the accident because although it didn't determine it was a bicycle/person until "too late" ... it had determined with plenty of time that it was maybe a car, maybe a bike.
    - beerandt 7y ago
      If it's having a hard time both identifying an object, as well as measuring it's movement, there's not really any reason it should understand that all those separate data points are the same object. That is, it doesn't really matter if the object history is "deleted" or not; if it can't associate a new data point with a previous history (by identification or predicted position), the practical result is the same as if there is no object history. This could be a result of using velocity based tracking, which I don't know that Uber uses, but is a fairly standard method, as it's what raw GPS measurements are based on.
Barrin92 7y ago
it speaks to a general problem of ML systems. Real-world problems are open-ended and a system that cannot reason about what it sees but merely applies object classification is completely clueless and won't reach a level of fidelity that is needed for safety. I'm increasingly convinced that virtually every unstructured problem in the physical world is an AI-hard problem and we won't be seeing fully autonomous driving for decades.
- gdulli 7y ago
  We as humans possess some skills that are so profoundly important but also so subtle that we don't even recognize them as skills. And excessive optimism about AI is a lack of recognition of how fundamental those skills are to our navigation of the world (both figuratively and literally.)
georgeecollins 7y ago
> with 0.2 seconds left before impact, the car sounded an audio alarm It takes about 300ms for your brain to react to unexpected stimulus, so the alarm is useless in this case. Sad.
- mrguyorama 7y ago
  The alarm's entire purpose is to shift the blame to the engineer in the driver's seat
sharkmerry 7y ago
Sounds like they fixed that post-crash.. Handling of Emergency Situations. ATG changed the way the ADS manages emergency situations (as described in section 1.6.2) by no longer implementing action suppression. The updated system does not suppress system response after detection of an emergency situation, even when the resolution of such situation—prevention of the crash—exceeds the design specifications. In such situations, the system allows braking even when such action would not prevent a crash; emergency braking is engaged to mitigate the crash. ATG increased the jerk (the rate of deceleration) limit to 20 m/s3 https://dms.ntsb.gov/public/62500-62999/62978/629713.pdf https://dms.ntsb.gov/public/62500-62999/62978/629713.pdf .
- NoodleIncident 7y ago
  Too bad they didn't do that after any of the previous 33 times it crashed into a vehicle
  - sharkmerry 7y ago
    I agree. Just pointed out that it is patched now...or rather, should be.
  - ErikCorry 7y ago
    There's lots of bad stuff in this story without making up new stuff. Those 33 times were other vehicles striking the Uber vehicle, rather than vice versa. There was one time where the Uber vehicle struck a stationary bicyle stand that was in the roadway.
    - lovehashbrowns 7y ago
      Is the story wrong? This is what it says: "In these 37 incidents, all of the robo-vehicles were driving in autonomous mode, and in 33, self-driving cars crashed into other vehicles." This is saying the self-driving cars crashed into other vehicles.
      - waiseristy 7y ago
chooseaname 7y ago
>> 5.6 seconds before impact, it classified her as a vehicle. Then it changed its mind to “other,” then to vehicle again, back to “other,” then to bicycle, then to “other” again, and finally back to bicycle. The system should have started applying brake at this point. If a 3500lb vehicle can't decide what it is about to impact, it needs to slow down (to a stop if necessary). > That triggered what Uber called “action suppression,” in which the system held off braking for one second This is borderline criminal negligence. > with 0.2 seconds left before impact, the car sounded an audio alarm, and Vasquez took the steering wheel, disengaging the autonomous system. Nearly a full second after striking Herzberg, Vasquez hit the brakes. Why were there no alarms going off at 5.6 seconds when the vehicle was confused!!!?? SMH. This is just ... I'm flabbergasted.
- gdulli 7y ago
  > The system should have started applying brake at this point. If a 3500lb vehicle can't decide what it is about to impact, it needs to slow down (to a stop if necessary). I suspect there's a big problem in the other direction, too. If the system starts to brake every time it thinks it might need to, it will happen all the time. It might see false positives like this all the time and that's why it doesn't act on them right away. If it's (appropriately) conservative about starting to brake it will be braking/slowing all the time. If it's not conservative, people or cars will occasionally get hit. The former could make people uncomfortable or carsick or create some subtler danger by stopping short needlessly. The latter might mostly work and mostly not kill people, except for when it does.
  - fmpwizard 7y ago
    If we apply this to a human, let's say I'm getting old and my eyes aren't as good any more, I don't just keep driving like nothing changed, I need to find a way to see better, because the risk of getting into an accident is higher. If the car's system cannot tell what an object is, you don't just assume it's going to be ok, you need to either get better sensors or find some other real solution.
  - trhway 7y ago
    >If the system starts to brake every time it thinks it might need to, it will happen all the time. It might see false positives like this all the time and that's why it doesn't act on them right away. if your car always sees empty space ahead as non-empty space, you probably shouldn't let the car on the road until you fix that. Once you fixed that, if the car sees that the space ahead is non-empty, even it can't classify it, it should slow down well before and warn the driver and continue with the braking if the driver is asleep/watching youtube. It is AZ, there is no rain nor snowflakes falling which would mislead the lidar. An object ahead - slow down and stop if you can't navigate it. Presenting it as the AI-hard issue of misclassification is just an attention misdirection from and a whitewhashing/laundering of the foundational issue of knowingly letting car on the road with missing basic safety level of "don't hit objects in front of the car". Similar to the Boeing blaming 737 MAX crashes on the failed sensor.
ars 7y ago
> " and Vasquez took the steering wheel, disengaging the autonomous system" > And then top it off with systemic issues around the backup driver not actually being ready to react. It's even worse than that! Once the human does take the wheel the computer stops doing anything. So from when the human is alerted and grabs the wheel, until the human can react, the car isn't even slowing down! That's like the worst of both worlds.
johnpowell 7y ago
Reminds me a bit of "Cisco Fixes RV320/RV325 Vulnerability by Banning “curl” in User-Agent".
- rrdharan 7y ago
  Wow. For anyone else who missed this: https://www.bleepingcomputer.com/news/security/cisco-botches-fix-for-rv320-rv325-routers-just-blocks-curl-user-agent/ https://www.bleepingcomputer.com/news/security/cisco-botches...
- deleted 7y ago
  [deleted]
bumby 7y ago
I would like to see the failure-mode-effects-analysis (FMEA) that identified "action suppression" as a means of mitigating a nuisance fault on a safety critical system. And understand why the designers felt this was okay...(Assuming of course, this was the actual reason for the delay. They may have a legitimate reason?) I hope it's not the case that the hazard analysis stated that the human in the loop was adequate no matter what haywire thing the software did.
xlc0212 7y ago
This is ridiculous. I already understand many seemly critical software are just unsafe / insecure but people actually running this without multiple layers of safety net mechanisms on a high speed machine that can, and did kill people? The backup driver is one broken safety net, and there are no more working security redundancy?
Slartie 7y ago
> > That triggered what Uber called “action suppression,” in which the system held off braking for one second > A hardcoded 1 second delay during a potential emergency situation. Horrifying. Also laughable, if it wasn't so horrifying. The self driving car evangelists always argue how much faster their cars could react than humans. It's basically their main selling point and the reason why these things ought to be so much safer than humans. Sorry, but I as a human don't have a one second delay built in. That's an absurdly slow reaction time for which I would have to be seriously drunk to not beat it.
- deleted 7y ago
  [deleted]
- shadowgovt 7y ago
  There's research on this topic, and you'd be surprised. The average human apparently has a 2.3 second delay to unexpected, interruptive stimulus while driving (https://copradar.com/redlight/factors/IEA2000_ABS51.pdf https://copradar.com/redlight/factors/IEA2000_ABS51.pdf). We almost never perceive it as such because we tend to measure our own reaction times from the point we are conscious of stimulus to the point we take willful action to respond to it, but the hard numbers appear to show that critical information can take 1+ seconds to percolate to conscious observation (remember, the brain spends a lot of time figuring out what in the soup of sensory nonsense is worthy of attention).
  - Slartie 7y ago
    The critical part is that you need to compare apples to apples - in this case, the one second delay is from the point at which the car had a clear idea of there being an obstacle in its path until it would have started to apply the brakes. If you want to compare this to humans, you also need to remove the sequence of time during which the human identifies the potential obstacle as relevant and subsequently as something he would crash into. Whether this time is shorter or longer for humans is another question entirely (though the human intelligences' ability to deduce intent from behavior and forecast actions of other humans in traffic should give robocars a good challenge in that department as well). But in terms of raw reaction time after determining "I have to brake NOW", a human is definitely faster than one second.
Florin_Andrei 7y ago
> I bet they added it because the system kept randomly thinking something serious was going to happen for a few milliseconds when everything was going fine. Smoothing bugs out via temporal integration. The oldest trick in the book.
jacquesm 7y ago
> > 5.6 seconds before impact, it classified her as a vehicle. Then it changed its mind to “other,” then to vehicle again, back to “other,” then to bicycle, then to “other” again, and finally back to bicycle. That alone should have been ground for immediate cessation of operation until a driver could take over, for the system to be declared unworthy of operation on public roads until this problem was fixed. The differences between 'pedestrian with bicycle', 'vehicle' and 'bicycle' are so large that any system that wants to steer a vehicle should be able to tell the three apart at at least 50 yards of distance or even more. That is the reason why regular drivers have to pass an eye test before they are allowed behind the wheel. If you can't see (or understand what you are seeing) you should not drive..
pweezy 7y ago
It seems like Uber put in this "reaction delay" to prevent the cars from driving/maneuvering erratically (think excessive braking and avoidance turning). This, along with allowing the cars to drive on public roads at all before handling obvious concerns like pedestrians outside of crosswalks, is supposed to be balanced out by having a human ready to intervene and handle these situations. I think one of the biggest lessons here is about the difficulty of relying on humans to maintain attention while covering an autonomous vehicle. Yes, this particular driver was actively negligent by apparently watching videos when they should have been monitoring the road. But even a more earnest driver could easily space out after long hours of uneventful driving with no "events" or inputs. And that could be enough that their delay in taking over could lead to the worst. Certainly not defending the safety driver here - or Uber. But I think there's a bit of a paradox in that the better an AV system performs, and the more the human driver trusts it, the easier it is for that human to mentally disengage. Even if only subconsciously. This seems like a difficult problem to overcome, especially if AV development is counting on tracking driver interventions to further train the models for new, unexpected, or fringe driving situations.
- shadowgovt 7y ago
  We will never have any way to know whether an average attentive human would have correctly parsed this situation or would also have hit the unexpected pedestrian in the middle of the street at night, but it's worth remembering that trying to make broad assessments of self-driving technology from this one accident is reasoning from a single data point. One advantage the self-driving cars have over a human driver is that NTSB and Uber can yank the memory and replay the logs to see what went wrong, correct the problem, and push the correction to the next generation of vehicles. That's not a trick you can pull off with our current fleet of human drivers, unfortunately(1). (1) This is not a universal problem with human operators, per se... The airline industry has a great culture of observing air accidents and learning from them as a responsibility of individual pilots. We don't have a similar process for individual drivers, and there are far, far more car crashes than air crashes so the time commitment would be impractical at 100% of accidents.
  - babesh 7y ago
choppaface 7y ago
From a year ago [1], engineers said jaywalker detection wasn't there: > Employees also said the car was not always able to predict the path of a pedestrian. The brake inhibition was very intentional and was the result of in-fighting as well as engineers trying to make the Dara demo: > Two days after the product team distributed the document discussing "rider-experience metrics" and limiting "bad experiences" to one per ride, another email went out. This one was from several ATG engineers. It said they were turning off the car's ability to make emergency decisions on its own like slamming on the brakes or swerving hard. ... > The subtext was clear: The car's software wasn't good enough to make emergency decisions. And, one employee pointed out to us, by restricting the car to gentler responses, it might also produce a smoother ride. ... > A few weeks later, they gave the car back more of its ability to swerve but did not return its ability to brake hard. And then they hit Herzberg. The UberATG leaders who made it through the Dara / Softbank demo likely vested (or are slated to vest) millions of dollars. [1] https://www.businessinsider.com/sources-describe-questionable-decisions-and-dysfunction-inside-ubers-self-driving-unit-before-one-of-its-cars-killed-a-pedestrian-2018-10 https://www.businessinsider.com/sources-describe-questionabl...
deleted 7y ago
[deleted]
sandworm101 7y ago
>> try to avoid a bicycle, in principle, instead of blindly gliding into it while hoping for the best. Better question than why did this happen: How often do these cars "see" bicycle and decide to glide on by. How often do they seeing things horribly incorrectly and we are all just lucky nothing happens.
unbalancedevh 7y ago
> A hardcoded 1 second delay during a potential emergency situation. Horrifying. As a controls engineer in the automotive industry, I can tell you that a 1-second delay for safety-critical systems is not atypical. The expectation is that the normal software avoids unsafe operation. Bounding "safe operation" is difficult, so if an excursion is detected, there's essentially a debounce period (up to 1 second) to let the normal software correct itself before override measures are taken. This helps prevent occasional random glitches or temporary edge cases from resulting in a system over-reaction, like applying the brakes or removing torque unnecessarily, that would annoy the driver and potentially cause unsafe operation themselves. Obviously there are still gaps with that approach. But there is supposed to be a driver in charge; and the intent is to prevent run-away unsafe behavior. It essentially boils down to due-diligence during development.
- nolok 7y ago
  To counter balance your point : the original Volvo emergency braking system in the car saw the crash and wanted to brake 1,3 seconds before it happened. So Volvo engineers didn't think at all like you do / say. Their system was 0,1 second faster than Uber at detecting it, 1,1 seconds faster if you factor in Uber active suppression, and it would have braked 2,1 seconds sooner than the Uber did. Why didn't it? Because Uber deactivated it's braking ability.

gambiting 7y ago

>>> 5.6 seconds before impact, it classified her as a vehicle. Then it changed its mind to “other,” then to vehicle again, back to “other,” then to bicycle, then to “other” again, and finally back to bicycle. This is exactly why I keep saying that autonomous vehicles are not 10 or even 20 years away. More like 50-100 years away if that. Same reason as to why famously a group of researchers in the 60s thought that solving computer recognition of objects would take few months at max, and yet in 2019 our best algorithms still think that a sofa covered in a zebra print is actually a zebra with 99% confidence. Had a human been actually paying attention to the road, I can bet they would start breaking/swearving as soon as they saw something, even if they weren't immediately 100% certain that it's a human - a computer won't until it's 99%+ certain, which is too risky assumption considering the state of visual recognition of objects.

SideburnsOfDoom 7y ago

> System can't decide what's happening. > At least the car would still try to avoid a bicycle, in principle, instead of blindly gliding into it while hoping for the best. "Don't know what it is, let's ram it." Never mind not detecting a pedestrian, that in itself is terrifyingly incompetent and negligent.

guenthert 7y ago

As I remember it, the driver got a lot of heat for fumbling with her phone (or 2nd computer?) right before the accident. I don't think however that 1.2s is a bad reaction time for a complex situation. Would it have killed the developers to make the car sound its horn when it gets into this absurd 1s "action suppression" mode?

Strilanc 7y ago
> the driver got a lot of heat for fumbling with her phone (or 2nd computer?) right before the accident Based on news stories I found, she was glancing at a television show on her phone [1]. > make the car sound its horn when it gets into this absurd 1s "action suppression" mode? If they added the suppression because there were too many false positives, that would just have resulted in the car honking at apparently arbitrary times. It's just converting the garbage signal from one form into another. It's still too noisy to be reliable. 1: https://www.azcentral.com/story/news/local/tempe/2019/03/17/one-year-after-self-driving-uber-rafaela-vasquez-behind-wheel-crash-death-elaine-herzberg-tempe/1296676002/ https://www.azcentral.com/story/news/local/tempe/2019/03/17/... Vasquez looked down 166 times when the vehicle was in motion, not including times she appeared to be checking gauges [...] In all, they concluded, Vasquez traveled about 3.67 miles total while looking away from the road. [...] starting at 9:16, Vasquez played an episode of “The Voice,” The Blind Auditions, Part 5, on her phone.
- mcphage 7y ago
  > If they added the suppression because there were too many false positives, that would just have resulted in the car honking at apparently arbitrary times. It's just converting the garbage signal from one form into another. It's still too noisy to be reliable. I love how they went from "our vision system is too unreliable to have warning signals every time it doesn't know what's in front of it" to "okay let's do it anyway but just not have warning signals". Like it didn't make them stop and think "well maybe basing a self-driving car off of this isn't a good idea".
  - EpicEng 7y ago

me_me_me 7y ago

I wonder why pick such vehicle to test this. Why not something with smaller mass to have less momentum on impact. (ie google car).

yifanl 7y ago
Actually yeah, can we not rig some kind of bicycle with the same sensors and test self-driving that way? The steering mechanism would have to be modified obviously, but surely steering is a trivial part of the problem compared to actually figuring out where to steer to?
- rtkwe 7y ago
  Part of path planning will involve vehicle dynamics, breaking, acceleration and steering response, and the envelope of the vehicle. All of those will heavily impact how a car should drive.
  - lovehashbrowns 7y ago
    But that's something the AI can learn fairly easily, isn't it? The difficulty in this case wasn't that the AI had issues figuring out the handling of the SUV, it's that it had issues detecting a pedestrian and a dangerous situation. You can still run into these issues on a bicycle, with a much lesser chance of killing people.
    - dkonofalski 7y ago
      This is totally true but the issue was more with the methodology of the detection rather than the detection itself. Regardless of the type of vehicle, the software wasn't good enough for real-world testing.
- fgvuyg 7y ago
  cycle dynamics is an unsolved control problem

Strilanc 7y ago

To rant a bit more about this one second delay thing. This reeks of a type of thinking where you are relying on other parts of the system to compensate. You might expect to hear things like "it's okay, the safety driver will catch it". Speaking for myself personally, this type of thinking comes very naturally. I like to come up with little proofs that a problem is handled by one part of a program, and then forget about that problem. But in my experience (which does not involve writing anything safety critical) this strategy kinda sucks at getting things right. Dependencies change, assumptions become invalid, holes in your intuitive proof become apparent, etc, etc, etc, and the thing falls over. If you are designing a safety critical system, something you really want to work, I don't think you should be thinking in the mode where each problem is handled by one part of the system. You need to be thinking in terms of defense in depth. When something goes wrong, many parts of the system should all be able to detect and correct the problem. And then when something bad does come up, and 9 out of 10 of those defensive layers each individually were sufficient to save the day so there was no disaster, you should go figure out what the hell went wrong in the tenth.

jeromebaek 7y ago
This. The right way to think is that each component, in parallel, have a chance of succeeding, so chance of total system failure is exponentially small in the number of components. Not: oh if this layer fails, the next one will catch it... which makes the chance of failure as high as the weakest link.
flowerlad 7y ago
> This reeks of a type of thinking where you are relying on other parts of the system to compensate. This is what Boeing did with Max. The airframe wasn’t stable in and of itself, and they relied on software to compensate. Terrible idea.
steelframe 7y ago
> Dependencies change, assumptions become invalid, holes in your intuitive proof become apparent, etc, etc, etc, and the thing falls over. I apply encryption to storage. I can't tell you how often people try to push back on encrypting storage with stories like, "But we have access controls and auditing in place. And when we have a deprovisioning process for our drives. Encryption is costly and redundant, so why should we do it?" Through the years I can recount several after-the-fact incidents where encryption ended up saving their bacon because of weird and entirely unanticipated events. One notable one was where a hypervisor bug caused memory to persist to an unintended location during suspend/resume, and the only reason customer data wasn't exposed in production was because the storage target was encrypted. In another case the "streams were crossed" when assigning virtual disks to virtual machines. The (older) base disk images weren't encrypted in that case, but because the newer machines were applying encryption in the backend before the blocks were exposed to the guest OS, the "unencrypted" disk content came across as garbage (plaintext was "decrypted," which with the algorithms we were using was equivalent to encrypting), again preserving the confidentiality of the original disk images. The concept of "belt and suspenders" is often lost on people when it comes to safety and security systems.

SilasX 7y ago

NoodleIncident 7y ago
The table goes into more detail. Each time the classification changed, the history for that object was essentially deleted; since there's only one data point, the system predicted that it would "continue" to stay stationary, even though the pedestrian was walking at a steady pace.
- rubicon33 7y ago
  That part, about deleting the history, confuses me. Why delete the history on a classification change? Shouldn't classifications be tiered? In this case, while the system was struggling to PERFECTLY classify the object, it was clearly thinking it was something that should be avoided (oscillating between car, bike, other). In this case, I would expect the system to keep it's motion history. IMO, this could have prevented the accident because although it didn't determine it was a bicycle/person until "too late" ... it had determined with plenty of time that it was maybe a car, maybe a bike.
  - beerandt 7y ago
    If it's having a hard time both identifying an object, as well as measuring it's movement, there's not really any reason it should understand that all those separate data points are the same object. That is, it doesn't really matter if the object history is "deleted" or not; if it can't associate a new data point with a previous history (by identification or predicted position), the practical result is the same as if there is no object history. This could be a result of using velocity based tracking, which I don't know that Uber uses, but is a fairly standard method, as it's what raw GPS measurements are based on.

Barrin92 7y ago

it speaks to a general problem of ML systems. Real-world problems are open-ended and a system that cannot reason about what it sees but merely applies object classification is completely clueless and won't reach a level of fidelity that is needed for safety. I'm increasingly convinced that virtually every unstructured problem in the physical world is an AI-hard problem and we won't be seeing fully autonomous driving for decades.

gdulli 7y ago
We as humans possess some skills that are so profoundly important but also so subtle that we don't even recognize them as skills. And excessive optimism about AI is a lack of recognition of how fundamental those skills are to our navigation of the world (both figuratively and literally.)

georgeecollins 7y ago

> with 0.2 seconds left before impact, the car sounded an audio alarm It takes about 300ms for your brain to react to unexpected stimulus, so the alarm is useless in this case. Sad.

mrguyorama 7y ago
The alarm's entire purpose is to shift the blame to the engineer in the driver's seat

sharkmerry 7y ago

Sounds like they fixed that post-crash.. Handling of Emergency Situations. ATG changed the way the ADS manages emergency situations (as described in section 1.6.2) by no longer implementing action suppression. The updated system does not suppress system response after detection of an emergency situation, even when the resolution of such situation—prevention of the crash—exceeds the design specifications. In such situations, the system allows braking even when such action would not prevent a crash; emergency braking is engaged to mitigate the crash. ATG increased the jerk (the rate of deceleration) limit to 20 m/s3 https://dms.ntsb.gov/public/62500-62999/62978/629713.pdf https://dms.ntsb.gov/public/62500-62999/62978/629713.pdf .

NoodleIncident 7y ago
Too bad they didn't do that after any of the previous 33 times it crashed into a vehicle
- sharkmerry 7y ago
  I agree. Just pointed out that it is patched now...or rather, should be.
- ErikCorry 7y ago
  There's lots of bad stuff in this story without making up new stuff. Those 33 times were other vehicles striking the Uber vehicle, rather than vice versa. There was one time where the Uber vehicle struck a stationary bicyle stand that was in the roadway.
  - lovehashbrowns 7y ago
    Is the story wrong? This is what it says: "In these 37 incidents, all of the robo-vehicles were driving in autonomous mode, and in 33, self-driving cars crashed into other vehicles." This is saying the self-driving cars crashed into other vehicles.
    - waiseristy 7y ago

chooseaname 7y ago

>> 5.6 seconds before impact, it classified her as a vehicle. Then it changed its mind to “other,” then to vehicle again, back to “other,” then to bicycle, then to “other” again, and finally back to bicycle. The system should have started applying brake at this point. If a 3500lb vehicle can't decide what it is about to impact, it needs to slow down (to a stop if necessary). > That triggered what Uber called “action suppression,” in which the system held off braking for one second This is borderline criminal negligence. > with 0.2 seconds left before impact, the car sounded an audio alarm, and Vasquez took the steering wheel, disengaging the autonomous system. Nearly a full second after striking Herzberg, Vasquez hit the brakes. Why were there no alarms going off at 5.6 seconds when the vehicle was confused!!!?? SMH. This is just ... I'm flabbergasted.

gdulli 7y ago
> The system should have started applying brake at this point. If a 3500lb vehicle can't decide what it is about to impact, it needs to slow down (to a stop if necessary). I suspect there's a big problem in the other direction, too. If the system starts to brake every time it thinks it might need to, it will happen all the time. It might see false positives like this all the time and that's why it doesn't act on them right away. If it's (appropriately) conservative about starting to brake it will be braking/slowing all the time. If it's not conservative, people or cars will occasionally get hit. The former could make people uncomfortable or carsick or create some subtler danger by stopping short needlessly. The latter might mostly work and mostly not kill people, except for when it does.
- fmpwizard 7y ago
  If we apply this to a human, let's say I'm getting old and my eyes aren't as good any more, I don't just keep driving like nothing changed, I need to find a way to see better, because the risk of getting into an accident is higher. If the car's system cannot tell what an object is, you don't just assume it's going to be ok, you need to either get better sensors or find some other real solution.
- trhway 7y ago
  >If the system starts to brake every time it thinks it might need to, it will happen all the time. It might see false positives like this all the time and that's why it doesn't act on them right away. if your car always sees empty space ahead as non-empty space, you probably shouldn't let the car on the road until you fix that. Once you fixed that, if the car sees that the space ahead is non-empty, even it can't classify it, it should slow down well before and warn the driver and continue with the braking if the driver is asleep/watching youtube. It is AZ, there is no rain nor snowflakes falling which would mislead the lidar. An object ahead - slow down and stop if you can't navigate it. Presenting it as the AI-hard issue of misclassification is just an attention misdirection from and a whitewhashing/laundering of the foundational issue of knowingly letting car on the road with missing basic safety level of "don't hit objects in front of the car". Similar to the Boeing blaming 737 MAX crashes on the failed sensor.

ars 7y ago

> " and Vasquez took the steering wheel, disengaging the autonomous system" > And then top it off with systemic issues around the backup driver not actually being ready to react. It's even worse than that! Once the human does take the wheel the computer stops doing anything. So from when the human is alerted and grabs the wheel, until the human can react, the car isn't even slowing down! That's like the worst of both worlds.

johnpowell 7y ago

Reminds me a bit of "Cisco Fixes RV320/RV325 Vulnerability by Banning “curl” in User-Agent".

rrdharan 7y ago
Wow. For anyone else who missed this: https://www.bleepingcomputer.com/news/security/cisco-botches-fix-for-rv320-rv325-routers-just-blocks-curl-user-agent/ https://www.bleepingcomputer.com/news/security/cisco-botches...
deleted 7y ago
[deleted]

bumby 7y ago

I would like to see the failure-mode-effects-analysis (FMEA) that identified "action suppression" as a means of mitigating a nuisance fault on a safety critical system. And understand why the designers felt this was okay...(Assuming of course, this was the actual reason for the delay. They may have a legitimate reason?) I hope it's not the case that the hazard analysis stated that the human in the loop was adequate no matter what haywire thing the software did.

xlc0212 7y ago

This is ridiculous. I already understand many seemly critical software are just unsafe / insecure but people actually running this without multiple layers of safety net mechanisms on a high speed machine that can, and did kill people? The backup driver is one broken safety net, and there are no more working security redundancy?

Slartie 7y ago

> > That triggered what Uber called “action suppression,” in which the system held off braking for one second > A hardcoded 1 second delay during a potential emergency situation. Horrifying. Also laughable, if it wasn't so horrifying. The self driving car evangelists always argue how much faster their cars could react than humans. It's basically their main selling point and the reason why these things ought to be so much safer than humans. Sorry, but I as a human don't have a one second delay built in. That's an absurdly slow reaction time for which I would have to be seriously drunk to not beat it.

deleted 7y ago
[deleted]
shadowgovt 7y ago
There's research on this topic, and you'd be surprised. The average human apparently has a 2.3 second delay to unexpected, interruptive stimulus while driving (https://copradar.com/redlight/factors/IEA2000_ABS51.pdf https://copradar.com/redlight/factors/IEA2000_ABS51.pdf). We almost never perceive it as such because we tend to measure our own reaction times from the point we are conscious of stimulus to the point we take willful action to respond to it, but the hard numbers appear to show that critical information can take 1+ seconds to percolate to conscious observation (remember, the brain spends a lot of time figuring out what in the soup of sensory nonsense is worthy of attention).
- Slartie 7y ago
  The critical part is that you need to compare apples to apples - in this case, the one second delay is from the point at which the car had a clear idea of there being an obstacle in its path until it would have started to apply the brakes. If you want to compare this to humans, you also need to remove the sequence of time during which the human identifies the potential obstacle as relevant and subsequently as something he would crash into. Whether this time is shorter or longer for humans is another question entirely (though the human intelligences' ability to deduce intent from behavior and forecast actions of other humans in traffic should give robocars a good challenge in that department as well). But in terms of raw reaction time after determining "I have to brake NOW", a human is definitely faster than one second.

Florin_Andrei 7y ago

> I bet they added it because the system kept randomly thinking something serious was going to happen for a few milliseconds when everything was going fine. Smoothing bugs out via temporal integration. The oldest trick in the book.

jacquesm 7y ago

> > 5.6 seconds before impact, it classified her as a vehicle. Then it changed its mind to “other,” then to vehicle again, back to “other,” then to bicycle, then to “other” again, and finally back to bicycle. That alone should have been ground for immediate cessation of operation until a driver could take over, for the system to be declared unworthy of operation on public roads until this problem was fixed. The differences between 'pedestrian with bicycle', 'vehicle' and 'bicycle' are so large that any system that wants to steer a vehicle should be able to tell the three apart at at least 50 yards of distance or even more. That is the reason why regular drivers have to pass an eye test before they are allowed behind the wheel. If you can't see (or understand what you are seeing) you should not drive..

pweezy 7y ago

It seems like Uber put in this "reaction delay" to prevent the cars from driving/maneuvering erratically (think excessive braking and avoidance turning). This, along with allowing the cars to drive on public roads at all before handling obvious concerns like pedestrians outside of crosswalks, is supposed to be balanced out by having a human ready to intervene and handle these situations. I think one of the biggest lessons here is about the difficulty of relying on humans to maintain attention while covering an autonomous vehicle. Yes, this particular driver was actively negligent by apparently watching videos when they should have been monitoring the road. But even a more earnest driver could easily space out after long hours of uneventful driving with no "events" or inputs. And that could be enough that their delay in taking over could lead to the worst. Certainly not defending the safety driver here - or Uber. But I think there's a bit of a paradox in that the better an AV system performs, and the more the human driver trusts it, the easier it is for that human to mentally disengage. Even if only subconsciously. This seems like a difficult problem to overcome, especially if AV development is counting on tracking driver interventions to further train the models for new, unexpected, or fringe driving situations.

shadowgovt 7y ago
We will never have any way to know whether an average attentive human would have correctly parsed this situation or would also have hit the unexpected pedestrian in the middle of the street at night, but it's worth remembering that trying to make broad assessments of self-driving technology from this one accident is reasoning from a single data point. One advantage the self-driving cars have over a human driver is that NTSB and Uber can yank the memory and replay the logs to see what went wrong, correct the problem, and push the correction to the next generation of vehicles. That's not a trick you can pull off with our current fleet of human drivers, unfortunately(1). (1) This is not a universal problem with human operators, per se... The airline industry has a great culture of observing air accidents and learning from them as a responsibility of individual pilots. We don't have a similar process for individual drivers, and there are far, far more car crashes than air crashes so the time commitment would be impractical at 100% of accidents.
- babesh 7y ago

choppaface 7y ago

From a year ago [1], engineers said jaywalker detection wasn't there: > Employees also said the car was not always able to predict the path of a pedestrian. The brake inhibition was very intentional and was the result of in-fighting as well as engineers trying to make the Dara demo: > Two days after the product team distributed the document discussing "rider-experience metrics" and limiting "bad experiences" to one per ride, another email went out. This one was from several ATG engineers. It said they were turning off the car's ability to make emergency decisions on its own like slamming on the brakes or swerving hard. ... > The subtext was clear: The car's software wasn't good enough to make emergency decisions. And, one employee pointed out to us, by restricting the car to gentler responses, it might also produce a smoother ride. ... > A few weeks later, they gave the car back more of its ability to swerve but did not return its ability to brake hard. And then they hit Herzberg. The UberATG leaders who made it through the Dara / Softbank demo likely vested (or are slated to vest) millions of dollars. [1] https://www.businessinsider.com/sources-describe-questionable-decisions-and-dysfunction-inside-ubers-self-driving-unit-before-one-of-its-cars-killed-a-pedestrian-2018-10 https://www.businessinsider.com/sources-describe-questionabl...

deleted 7y ago

[deleted]

sandworm101 7y ago

>> try to avoid a bicycle, in principle, instead of blindly gliding into it while hoping for the best. Better question than why did this happen: How often do these cars "see" bicycle and decide to glide on by. How often do they seeing things horribly incorrectly and we are all just lucky nothing happens.

unbalancedevh 7y ago

> A hardcoded 1 second delay during a potential emergency situation. Horrifying. As a controls engineer in the automotive industry, I can tell you that a 1-second delay for safety-critical systems is not atypical. The expectation is that the normal software avoids unsafe operation. Bounding "safe operation" is difficult, so if an excursion is detected, there's essentially a debounce period (up to 1 second) to let the normal software correct itself before override measures are taken. This helps prevent occasional random glitches or temporary edge cases from resulting in a system over-reaction, like applying the brakes or removing torque unnecessarily, that would annoy the driver and potentially cause unsafe operation themselves. Obviously there are still gaps with that approach. But there is supposed to be a driver in charge; and the intent is to prevent run-away unsafe behavior. It essentially boils down to due-diligence during development.

nolok 7y ago
To counter balance your point : the original Volvo emergency braking system in the car saw the crash and wanted to brake 1,3 seconds before it happened. So Volvo engineers didn't think at all like you do / say. Their system was 0,1 second faster than Uber at detecting it, 1,1 seconds faster if you factor in Uber active suppression, and it would have braked 2,1 seconds sooner than the Uber did. Why didn't it? Because Uber deactivated it's braking ability.