31 ms·

Uber vehicle reportedly saw but ignored woman it struck

337 points by runesoerensen 8y ago

NegativeLatency 8y ago
Didn't that car also have a builtin automatic braking system that was disabled?
- jsight 8y ago
  I think so. But I'm not sure why that would be a bad thing. The Uber built system was designed to fully replace such systems.
  - nasmorn 8y ago
    Hubris is the reason it is bad.
  - leggomylibro 8y ago
    As long as Uber's system logged which automatic brake was triggered - or simply failed to log that its own braking system stepped in - it could have saved a life without impacting the data gathered. They didn't put a tarp over the windshield; why get rid of other proven safety features?
    - carlosdp 8y ago
      You don't generally want multiple autonomous control systems in a single vehicle competing with each other for control. There are scenarios where the less-sophisticated collision avoidance radar could do the unsafe thing where the self-driving system has a more holistic plan. Basically, there's no reason the self-driving tech shouldn't have been way more effective than the collision avoidance radar other than Uber royally screwed up. The LIDAR and other sensors should have (and by this article, likely did) detected the woman long before the vehicle was in striking distance, much longer range than the forward radar.
      - jameshart 8y ago
        That argument applies equally to human drivers competing for control of a vehicle with automated braking systems. Are there not also scenarios where the less-sophisticated collision avoidance radar could do the unsafe thing where the human has a more holistic plan? If not, why not?
      - js8 8y ago
        > There are scenarios where the less-sophisticated collision avoidance radar could do the unsafe thing where the self-driving system has a more holistic plan. Like, for example, this plan? https://www.youtube.com/watch?v=od5YG6kBrJs https://www.youtube.com/watch?v=od5YG6kBrJs I believe the whole thing will require better conceptualization of something that we call crew resource management for human teams. That is, how do you respond if you have conflicting information?
      - deleted 8y ago
        [deleted]
      - vkou 8y ago
        > You don't generally want multiple autonomous control systems in a single vehicle competing with each other for control. And yet, every modern car has two such systems, competing for control, all the time. 1. The car's safety systems. 2. The human driver.
tlb 8y ago
The disturbing thing about this crash is that it's such a basic thing to test on a test track. A pedestrian walking in front at night, with nothing else unusual happening, should be on the list of scenarios to get 5 9s reliability on before taking it out in public. The things you have to test in public are the complex real-world interactions, like these: https://medium.com/kylevogt/why-testing-self-driving-cars-in-sf-is-challenging-but-necessary-77dbe8345927 https://medium.com/kylevogt/why-testing-self-driving-cars-in...
- retSava 8y ago
  That, and the inattentive safety driver. It's quite infuriating really, letting lose a ton and a half of fast-moving mass without even the decency of looking where it's going. There's really no excuse.
  - CobrastanJorji 8y ago
    Unfortunately I think it's one of those things that is both inexcusable and guaranteed to happen. It absolutely makes sense to put a human in a car you're testing and don't have full confidence in yet but are sending into real traffic. Not doing so would be ludicrously negligent. However, humans simply don't maintain focus for a long period of time without some sort of extreme, years-long training. The ability of monks to concentrate on a single thought for hours is correctly seen as very, very difficult. There's no way any regular person could deal well with a scenario like "sometime in the next 8 hours, you may or may not need to smash this brake pedal with a second's notice, but other than that you have no task." That's one of the things that most worries me about "almost there" systems like Tesla's. That said, if you've decided to attempt this anyway, checking your phone is definitely not okay.
    - deleted 8y ago
      [deleted]
    - Rjevski 8y ago
      However, if the car saw the pedestrian, even as a false positive (presumably from far enough away to have enough time to brake/avoid it), I think it should’ve alerted the driver. The driver would’ve most likely started paying attention and avoided the crash.
assblaster 8y ago
The question I want to know is: Did Uber doctor the camera video to make the road seem darker than it actually was? If you recall, there was initially a doubt that Uber was at fault because the streets appeared extremely dark, but I think they were buying time to delay the outage at incompetent Uber. The police initially didn't fault Uber based on this video.
- jakelazaroff 8y ago
  That was a bad call on the part of the police. You're only allowed to drive as fast as conditions permit. If it's dark enough that you can't avoid an obstacle by the time your headlights illuminate it, you need to slow down.
  - hazardmat 8y ago
    That's not really true. Thousands of deer get killed on the road every year, because deer aren't smart enough not to be an obstacle when it's dark. Do we therefore change all the speed limits at night? No. It's a driving risk taken, just like any other.
    - favorited 8y ago
      The point isn't that speed limits need to change, it's that it is incumbent upon the driver to slow down if environmental conditions prevent them from traveling safely at the speed limit.
      - hazardmat 8y ago
        What is that supposed to mean? You can't have 100% safety at any speed. If it really is incumbent upon the driver to always drive at night in such a way that no collision can occur at all, then why not change the speed limit? My whole point is that it's not the driver's responsibility to account for every possible scenario. It can't be.
        deleted 8y ago
mtgx 8y ago
> This puts the fault squarely on Uber’s doorstep, though there was never much reason to think it belonged anywhere else. Are you kidding me? At least 80% of the comments I saw online the following week were blaming the woman. It seems to me like Uber's self-driving system is absolute trash. But they were in a hurry to "go live" with it, because they need it so that the company can become profitable ASAP. And thanks to the deregulation bill from last year (which was almost universally praised online), they could actually do that, too.
- atomical 8y ago
  I totally agree. There's a PR cover up going on.
- ajross 8y ago
  It's still the biker's "fault" in some sense. That same video would have 100% exonerated a human driver. There's no way that was a safe crossing. The fact that there was a clear automation fault also doesn't change the fact that traffic safety is everyone's job, including but not exclusively the computers.
  - telchar 8y ago
    You mean the video that showed the safety driver staring at a cell phone for 6 full seconds right up until nearly the moment of impact? Maybe that would have exonerated the driver, but only to prove how perverse our driving laws are. But in my state the driver would have been convicted.
    - Rapzid 8y ago
      Do we know what she was looking at? I haven't read anything conclusive of that. What would the driver have been convicted of? Would you be willing to get second opinions from prosecutors in your state as to whether or not they would have persued a conviction of that crime?
      - telchar 8y ago
freditup 8y ago
I'm curious what degree of "cultural context" is built into automated cars - I feel that there has to be a lot of locale-specific techniques for driving. For example, in NYC, there's nothing at all uncommon about a pedestrian beginning to cross the street in front of you and approaching the area of your lane where you will shortly be. Pedestrians typically then stop about a foot away from your lane, let you drive past them, and then continue walking. If a car slammed on the brakes in this situation, it would likely cause more accidents than not braking would cause. On the contrary, in Tempe, a pedestrian starting to do the exact same thing as above is likely much more a case of them not realizing your car is coming at all, in which case slamming on the brakes is appropriate. This isn't to defend Uber, clearly their car did do the wrong thing in the Tempe situation (though I make no judgment on who was at fault). And clearly cars need to be able to handle the variety of unwritten, localized driving quirks different regions have. But it seems like a very non-trivial problem to do this correctly.
- GenerocUsername 8y ago
  And welcome to why self-driving cars will always be 3 years away. I think we should be targeting self driving for long-haul freeway scenarios where the rules are relatively simple and predictable. Trying to control for the infinite number of scenarios in cities is a nightmare which will ultimately bog down the development of any company which makes that its goal.
  - joncrane 8y ago
    > And welcome to why self-driving cars will always be 3 years away. It's also possible that human/pedestrian behavior will adapt to the consistent behavior of driverless cars.
    - splouk 8y ago
      Or jaywalking will become more strictly enforced with policing or physical barriers. https://www.strongtowns.org/journal/2018/4/2/automated-vehicles https://www.strongtowns.org/journal/2018/4/2/automated-vehic...
jacquesm 8y ago
Anybody still want to defend the police's claim within a day of the accident putting the blame on the woman that got killed?
- hazardmat 8y ago
  Yes. They said "preliminarily" the Uber driver was not at fault. Judging from the footage, that was reasonable. You can't put the blame off the woman, you can't just cross a road at night like that and expect to be noticed. Accidents like that happen all the time, it's only because of the self-driving car that this is newsworthy. Imagine if there wasn't any dashcam footage and this was a normal car with a distracted driver, without any other witnesses they would likely be off the hook.
  - jacquesm 8y ago
    A distracted driver has the option to tell the police they were distracted. Or are you of the opinion that since the woman was dead it didn't matter anymore?
    - hazardmat 8y ago
      First of all, in this scenario, the woman would have at least shared some of the blame. Second of all, what are the odds that a human who just killed a woman would admit even to themselves that they were the one to blame in such an accident? Practically zero. Cognitive dissonance won't allow it. In their minds, they weren't distracted, at least not enough to take any of the blame.
      - jacquesm 8y ago
        > First of all, in this scenario, the woman would have at least shared some of the blame. Possibly. But a distracted driver is definitely at fault, no matter what. Driving is a full time occupation. > Second of all, what are the odds that a human who just killed a woman would admit even to themselves that they were the one to blame in such an accident? Pretty good. Most people are honest, even in a bad situation. > Practically zero. You speak for yourself. > Cognitive dissonance won't allow it. In their minds, they weren't distracted, at least not enough to take any of the blame. Well, you probably have your reasons for writing this under a novelty account. But it's clear that when you are at fault it is best for everybody involved to recognize that fact, even if it means discomfort for you. I get that in todays climate it's what you can get away with, but I don't think that strategy will hold in the longer term. It's way each and every lawsuit is appealed to the highest courts because nobody bothers to just admit their errors and it's why when there is a car accident the perpetrators will sue the victims. But that's wrong on many levels and I refuse to subscribe to that view.
w_t_payne 8y ago
I think that I am qualified to say that cost-effective testing is and will continue to be the barrier to self-driving car adoption. A big part of the solution to this problem is design-for-test; and the adoption and tight integration of a wide range of (quite conventional but sadly often overlooked) testing and systems engineering standards and processes.
toast0 8y ago
> We’re actively cooperating with the NTSB in their investigation. Out of respect for that process and the trust we’ve built with NTSB, we can’t comment on the specifics of the incident. Hey look, somebody knows how to follow the script! Uber may be grossly negligent, but at least they respect the process unlike Tesla.
- TillE 8y ago
  It's actually the rest of that statement which shocks me a bit: > In the meantime, we have initiated a top-to-bottom safety review of our self-driving vehicles program, So they actually think they can continue to work on self-driving vehicles after this, and not scrap the program entirely. That's...not what I expected. No municipality in the country should allow Uber to test on their streets after they killed someone, almost certainly because of their recklessly fast development. That's not something you come back from by promising to Do Better, especially not when there are more than a few competitors already way ahead of you.
  - lhorie 8y ago
    > No municipality in the country should allow Uber to test on their streets after they killed someone This is pitchfork mentality. Many industries have to deal with unintentional deaths. Car manufacturers sometimes have to deal with deaths resulting from hardware failure. Food companies sometimes have to deal with recalls due to food poisoning deaths. If the response to a death is to ban all further production work, eventually nothing is ever going to get done because it's inherently impossible to be perfect all the time. It sucks that it had to come to someone's death, but the rest of the statement said: > we have brought on former NTSB Chair Christopher Hart to advise us on our overall safety culture so it sounds like Uber is at least trying in earnest to put the house in order.
    - cimmanom 8y ago
      Or a cynic could interpret that as "we've hired a crony of the people who will be judging us so he can lobby them to let us off easy".
SilasX 8y ago
Sorry for the snark, but there's a joke in here about: "Uber business development saw but ignored laws it broke."
- Udik 8y ago
  There is without doubt a grotesque irony in this story. Uber stands as a perfect symbol of the reckless capitalism of the 21st century: invests hundreds of millions in building a self-driving monster in the shape of a 2 ton luxury SUV, then puts behind its wheel- as a "safety driver"- a former convict for attempted robbery. And the capitalism-fueled metal golem immediately crushes a homeless woman pushing a humble bicycle. I mean, it could have been written as a satire of our times. Except, it really happened.
- hughes 8y ago
  Snark aside, false-positive rejection will be an aspect of any lawful autonomous vehicle. Incorrect false-positive rejection will be a reality, and will have to exist within lawful operation.
thegambit 8y ago
It seems like it was actually Uber who disabled the safety settings that were there to prevent this according to Volvo. "Uber Technologies Inc. disabled the standard collision-avoidance technology in the Volvo SUV that struck and killed a woman in Arizona last week, according to the auto-parts maker that supplied the vehicle’s radar and camera." https://www.bloomberg.com/news/articles/2018-03-26/uber-disabled-volvo-suv-s-standard-safety-system-before-fatality https://www.bloomberg.com/news/articles/2018-03-26/uber-disa...
- nmca 8y ago
  This whole line of reasoning frustrates me somewhat. The story of "Uber Disabled the Safety System" paints Uber out to be negligent for doing so. Outside of consideration of all the other facts (which unlike this one, do indeed suggest that Uber are to blame), this element of the story is not indicative of negligence. Of course they disabled the existing safety system - they were aiming to build a new, safer one that could operate without reliance on or interaction with the existing one. That (again, on its own) seems like a totally reasonable engineering call.
  - linker3000 8y ago
    Seriously!? The responsible position would be to retain the existing system in a passive standby mode, capable of overriding the active 'under test' one in the event certain safety/detection thresholds were exceeded.
    - lhorie 8y ago
      IIRC someone had said that there are physical limitations to what one can actually do (namely, having to physically unplug the connector for one safety system in order to be able to add your own at all)
      - telchar 8y ago
        Sure, if you want to move fast and disrupt yada yada. They shouldn't be putting hack-job self-driving cars on public roads. Either do some engineering so that they keep the backup system in place or keep it on a test track. This is Chernobyl-like irresponsibility.
sharemywin 8y ago
Is this a binary decision? I normally slow down when I see a something at the side of the road.
wilburTheDog 8y ago
If I understand it right, this is the biggest blind spot of all the automated driving systems. The software just ignores stationary objects because the sensors don't have the resolution to determine whether they are in front of the car or on the side of the road. And if it didn't ignore stationary objects the car would come to a halt at every garbage can, mailbox, and bush along the road. Until they can work lidar into the system this will not be preventable. And, correct me if I'm wrong, but I think the vast majority of accidents that were the fault of the self-driving car happened because it hit something stationary in the road (or off to the side of the road, in one case I recall). Since I've learned about this blind spot in these system I no longer trust them. I think everyone with a self-driving car should understand this. But maybe it's not in the interest of the manufacturer to explain it, or people would demand they fix the problem, which they cannot do just yet.
- Symmetry 8y ago
  That wasn't the case at all for this collision. As the article says, the sensors had detected the woman but the information had been disregarded. And the moving object thing is exclusive to radar, vision systems don't suffer from it though vision systems have their own problems.
  - wilburTheDog 8y ago
    The article linked mentioned "B was the problem", meaning it ignored the sensor input. Another article(1) about it mentions that Uber had reduced the number of LIDAR sensors from five to just one rooftop sensor. The president of the company that builds the sensors even said you would need side sensors to see pedestrians, especially at night. But LIDAR is expensive. So it seems that, like Tesla, they were trying to rely on radar for that kind of thing. But that does leave the blind spot(2) I mentioned. Given that I don't see how this case is so different. Though I still don't understand how, specifically, that blind spot problem works. They say when the car in front of you changes lanes and a stopped truck is now in your lane you'll hit it. What if no car changed lanes in front of you? Would you still hit a stopped truck in your lane? The guy whose Tesla hit the bridge column seems to tell me yes, but I'm not sure. 1. https://www.theregister.co.uk/2018/03/28/uber_selfdriving_death_may_have_been_due_to_lidar_blind_spot/ https://www.theregister.co.uk/2018/03/28/uber_selfdriving_de... 2. https://www.wired.com/story/tesla-autopilot-why-crash-radar/ https://www.wired.com/story/tesla-autopilot-why-crash-radar/
Animats 8y ago
I'm looking forward to seeing the full NTSB report. This hints that Uber's mindset is focused on "obstacles", not "flat road". The first job of automated driving is to drive only on flat surfaces. Doesn't matter why it's not flat. Then you worry about where to go. That's how the off-road DARPA Grand Challenge vehicles had to work. "Flat road" is a pure geometry thing. Not much AI needed. On road, non-flat road is rare, and you don't have to worry much about going off cliffs, rocks in the road, and such. So it's tempting to focus on "obstacles" to be tallied and classified. Tesla definitely has an "obstacle" focus, and a limited class of obstacles considered. Waymo profiles the ground. Looks like Uber had the "obstacle" focus. If you have a "flat road" detection system, things the obstacle detector doesn't understand get stopped for or bypassed. So the vehicle isn't vulnerable to this flaw. There's a higher false alarm rate, though. And a non false alarm problem. A piece of trash on the road is likely to result in a slowdown and careful avoidance, not a bump as the car rolls over it. Still, better to take the safe approach unless you're on a freeway and the cars ahead of you just got past the obstacle successfully. The Udacity self driving car "nanodegree" trains people to build obstacle detectors and classifiers. That may be a problem.
- krschultz 8y ago
  Is another way of framing this: "flat road" => assume you can't continue until you prove it's safe "obstacle detection" => assume you can continue unless there is something blocking the current path The bit of autonomous vehicle work that I did started from a set of potential trajectories that were then pruned by obstacle detection. So we always assumed you could continue moving forward unless every potential trajectory was pruned. We ignored the cliff problem as beyond our scope.
- dotancohen 8y ago
  > The first job of automated driving is to drive only on flat surfaces. Doesn't matter why it's not flat. Then you worry about where to go. The old adage applies. Aviate, Navigate, Communicate. In that order.
jessaustin 8y ago
ISTM that HN policy would replace TFA with the link cited in TFA: https://www.theinformation.com/articles/uber-finds-deadly-accident-likely-caused-by-software-set-to-ignore-objects-on-road https://www.theinformation.com/articles/uber-finds-deadly-ac... ...although, there is a paywall, so maybe not.
- jessriedel 8y ago
  Not just any paywall either. It's $40/month.
deleted 8y ago
[deleted]
mcr1983 8y ago
Uber's ultimately at fault because it was their car. But the woman driving should be arrested for vehicular manslaughter. Uber knew its cars weren't autonomous, that's why they had a driver to take over when the car screws up. But she was texting while she should've been ready to take control of the vehicle and hit a pedestrian any driver that was paying attention would've easily avoided.
- bkor 8y ago
  > Uber's ultimately at fault because it was their car. Uber while hiring should've specifically looked for people who can pay attention for hours on end. Basically like a train driver. Further, they should review if people pay attention and have a system in place. She was employed by Uber. If you make a huge error while working for someone, often it's the employer who is responsible. There's way too much which should've been done which wasn't to just blame this driver.
bsaul 8y ago
i haven't read much about the observation that driving = communicating. aka : whenever you drive, you send and receive information from all the environment and human, establishing sometimes what could be similar to a dialogue. Doesn't this implie that autonomous driving won't be solved until we're able to have "true" personnal assistant ? The message in driving are simpler than with a regular voice communication, but the objects and concepts visually present seem just as diverse.
chiefalchemist 8y ago
"Saw" doesn't seem like the right word. Granted, apparently, the sensors picked something up. What it failed to do is recognize and categorize; and from there properly take action. Given the failure of the recognition + categorization system, it would be interesting to know what it thought it saw. Was it filed under IDK, a plastic bag blowing in the wind, what?
someonewhocar3s 8y ago
One of the final tests, mandatory for regulatory acceptance: Produce accident on target individual within timeframe
deleted 8y ago
[deleted]
m3kw9 8y ago
What’s worse, it saw it but misinterperated or sensors has a blind spot and missed something?
stillsut 8y ago
The exact circumstances of the struck pedestrian likely violated many [possibly curde] bayesian priors for what to consider a positive collision threat: A rogue pedestrian at an unflagged portion of road, at an odd time of night, on a road that doesn't usually have pedestrians, with a bike moving perpendicular across the road (instead of parallel along it). Each one of these had been learned to be a high false positive to true negative ratio. The product of these rarely encountered events (even independently) allowed a higher level algorithm to "score" the approaching unverified object through bayesian inference, below the level of a reasonable expectation of human collision. In a way, this system should never be expected to become overly reckless or feckless as compared with even top human drivers in the long run: close calls of this nature should be input to the system (perhaps deliberately through staged QA?) and added to the model of collision threat identification.
- dmix 8y ago
  That is the classic self-driving tradeoffs question, no? I believe rather than absolutes, the solution is her is properly identifying a serious crash causing injury was missed. Regardless if it was ID'd as human, it was still someone pushing a shopping cart full of bags across a highway intersection that could have been detected. Even if it wasn't human, and totally dangerous and improper crossing at night time on a busy highway, it should have stopped even for the drivers own safety.
- justicezyx 8y ago
  I am rather confused: Should the car always respond by slow down as the mandatary response when there is object in front? Are you implying that the algorithm would conclude that because it's unlikely that there is an object in that circumstance, it is ok to disapprove its own observation that there is some thing in front of the car? That sounds utterly nonsensical.
  - danenania 8y ago
    Exactly. I'd expect collision-prevention to have absolute priority over any other system. It should not be possible for any logic bug or combination of environmental factors to make the car run into a large object in its field of vision. There's no need for statistics here--an overriding 'don't run into stuff' rule will suffice. If there's truly no possible route to avoid some collision and therefore the car needs to decide which collision is preferred, then the statistics can kick in, but in this Uber scenario, it shouldn't have gotten that far. From the outside, it seems like a fundamental design issue.
jiveturkey 8y ago
i wonder how this info got out. i thought (learned this from tesla) that companies are not allowed to release info about accidents under NTSB investigation. which this one is.
dingo_bat 8y ago
> This puts the fault squarely on Uber’s doorstep, though there was never much reason to think it belonged anywhere else. > This is not good. The state of journalism saddens me.
erdo 8y ago
I'm wondering how much more attentive the emergency driver would have been if they were one of the developers. Sometimes I worry about how much trust non-developers assign to software (in all sorts of areas)
kalal 8y ago
Sounds like the departments are fighting over the responsibility of the accident. Correct me if I am wrong.
jlebrech 8y ago
Ethical dilemma, should a vehicle swerve to avoid an unavoidable pedestrian and potentially killing more people or should it ignore that person (not saying that what happened) Should a autonomous vehicle also prioritise it's own passenger over the safety of others, aka head on collision with a people carrier.
agumonkey 8y ago
it's dramatic but it's also a good reminder that "modern" "recent" wonder tech doesn't comes free. It needs proper examination. This will surely push toward more serious work.
dboreham 8y ago
Um...if I as a human see a plastic bag on the road ahead, I slow down to be able to carefully ascertain if it perhaps is a plastic bag full of nails. I don't drive full tilt over it.

337 points by runesoerensen 8y ago

NegativeLatency 8y ago

Didn't that car also have a builtin automatic braking system that was disabled?

jsight 8y ago
I think so. But I'm not sure why that would be a bad thing. The Uber built system was designed to fully replace such systems.
- nasmorn 8y ago
  Hubris is the reason it is bad.
- leggomylibro 8y ago
  As long as Uber's system logged which automatic brake was triggered - or simply failed to log that its own braking system stepped in - it could have saved a life without impacting the data gathered. They didn't put a tarp over the windshield; why get rid of other proven safety features?
  - carlosdp 8y ago
    You don't generally want multiple autonomous control systems in a single vehicle competing with each other for control. There are scenarios where the less-sophisticated collision avoidance radar could do the unsafe thing where the self-driving system has a more holistic plan. Basically, there's no reason the self-driving tech shouldn't have been way more effective than the collision avoidance radar other than Uber royally screwed up. The LIDAR and other sensors should have (and by this article, likely did) detected the woman long before the vehicle was in striking distance, much longer range than the forward radar.
    - jameshart 8y ago
      That argument applies equally to human drivers competing for control of a vehicle with automated braking systems. Are there not also scenarios where the less-sophisticated collision avoidance radar could do the unsafe thing where the human has a more holistic plan? If not, why not?
    - js8 8y ago
      > There are scenarios where the less-sophisticated collision avoidance radar could do the unsafe thing where the self-driving system has a more holistic plan. Like, for example, this plan? https://www.youtube.com/watch?v=od5YG6kBrJs https://www.youtube.com/watch?v=od5YG6kBrJs I believe the whole thing will require better conceptualization of something that we call crew resource management for human teams. That is, how do you respond if you have conflicting information?
    - deleted 8y ago
      [deleted]
    - vkou 8y ago
      > You don't generally want multiple autonomous control systems in a single vehicle competing with each other for control. And yet, every modern car has two such systems, competing for control, all the time. 1. The car's safety systems. 2. The human driver.

tlb 8y ago

The disturbing thing about this crash is that it's such a basic thing to test on a test track. A pedestrian walking in front at night, with nothing else unusual happening, should be on the list of scenarios to get 5 9s reliability on before taking it out in public. The things you have to test in public are the complex real-world interactions, like these: https://medium.com/kylevogt/why-testing-self-driving-cars-in-sf-is-challenging-but-necessary-77dbe8345927 https://medium.com/kylevogt/why-testing-self-driving-cars-in...

retSava 8y ago
That, and the inattentive safety driver. It's quite infuriating really, letting lose a ton and a half of fast-moving mass without even the decency of looking where it's going. There's really no excuse.
- CobrastanJorji 8y ago
  Unfortunately I think it's one of those things that is both inexcusable and guaranteed to happen. It absolutely makes sense to put a human in a car you're testing and don't have full confidence in yet but are sending into real traffic. Not doing so would be ludicrously negligent. However, humans simply don't maintain focus for a long period of time without some sort of extreme, years-long training. The ability of monks to concentrate on a single thought for hours is correctly seen as very, very difficult. There's no way any regular person could deal well with a scenario like "sometime in the next 8 hours, you may or may not need to smash this brake pedal with a second's notice, but other than that you have no task." That's one of the things that most worries me about "almost there" systems like Tesla's. That said, if you've decided to attempt this anyway, checking your phone is definitely not okay.
  - deleted 8y ago
    [deleted]
  - Rjevski 8y ago
    However, if the car saw the pedestrian, even as a false positive (presumably from far enough away to have enough time to brake/avoid it), I think it should’ve alerted the driver. The driver would’ve most likely started paying attention and avoided the crash.

assblaster 8y ago

The question I want to know is: Did Uber doctor the camera video to make the road seem darker than it actually was? If you recall, there was initially a doubt that Uber was at fault because the streets appeared extremely dark, but I think they were buying time to delay the outage at incompetent Uber. The police initially didn't fault Uber based on this video.

jakelazaroff 8y ago
That was a bad call on the part of the police. You're only allowed to drive as fast as conditions permit. If it's dark enough that you can't avoid an obstacle by the time your headlights illuminate it, you need to slow down.
- hazardmat 8y ago
  That's not really true. Thousands of deer get killed on the road every year, because deer aren't smart enough not to be an obstacle when it's dark. Do we therefore change all the speed limits at night? No. It's a driving risk taken, just like any other.
  - favorited 8y ago
    The point isn't that speed limits need to change, it's that it is incumbent upon the driver to slow down if environmental conditions prevent them from traveling safely at the speed limit.
    - hazardmat 8y ago
      What is that supposed to mean? You can't have 100% safety at any speed. If it really is incumbent upon the driver to always drive at night in such a way that no collision can occur at all, then why not change the speed limit? My whole point is that it's not the driver's responsibility to account for every possible scenario. It can't be.
      - deleted 8y ago

mtgx 8y ago

> This puts the fault squarely on Uber’s doorstep, though there was never much reason to think it belonged anywhere else. Are you kidding me? At least 80% of the comments I saw online the following week were blaming the woman. It seems to me like Uber's self-driving system is absolute trash. But they were in a hurry to "go live" with it, because they need it so that the company can become profitable ASAP. And thanks to the deregulation bill from last year (which was almost universally praised online), they could actually do that, too.

atomical 8y ago
I totally agree. There's a PR cover up going on.
ajross 8y ago
It's still the biker's "fault" in some sense. That same video would have 100% exonerated a human driver. There's no way that was a safe crossing. The fact that there was a clear automation fault also doesn't change the fact that traffic safety is everyone's job, including but not exclusively the computers.
- telchar 8y ago
  You mean the video that showed the safety driver staring at a cell phone for 6 full seconds right up until nearly the moment of impact? Maybe that would have exonerated the driver, but only to prove how perverse our driving laws are. But in my state the driver would have been convicted.
  - Rapzid 8y ago
    Do we know what she was looking at? I haven't read anything conclusive of that. What would the driver have been convicted of? Would you be willing to get second opinions from prosecutors in your state as to whether or not they would have persued a conviction of that crime?
    - telchar 8y ago

freditup 8y ago

I'm curious what degree of "cultural context" is built into automated cars - I feel that there has to be a lot of locale-specific techniques for driving. For example, in NYC, there's nothing at all uncommon about a pedestrian beginning to cross the street in front of you and approaching the area of your lane where you will shortly be. Pedestrians typically then stop about a foot away from your lane, let you drive past them, and then continue walking. If a car slammed on the brakes in this situation, it would likely cause more accidents than not braking would cause. On the contrary, in Tempe, a pedestrian starting to do the exact same thing as above is likely much more a case of them not realizing your car is coming at all, in which case slamming on the brakes is appropriate. This isn't to defend Uber, clearly their car did do the wrong thing in the Tempe situation (though I make no judgment on who was at fault). And clearly cars need to be able to handle the variety of unwritten, localized driving quirks different regions have. But it seems like a very non-trivial problem to do this correctly.

GenerocUsername 8y ago
And welcome to why self-driving cars will always be 3 years away. I think we should be targeting self driving for long-haul freeway scenarios where the rules are relatively simple and predictable. Trying to control for the infinite number of scenarios in cities is a nightmare which will ultimately bog down the development of any company which makes that its goal.
- joncrane 8y ago
  > And welcome to why self-driving cars will always be 3 years away. It's also possible that human/pedestrian behavior will adapt to the consistent behavior of driverless cars.
  - splouk 8y ago
    Or jaywalking will become more strictly enforced with policing or physical barriers. https://www.strongtowns.org/journal/2018/4/2/automated-vehicles https://www.strongtowns.org/journal/2018/4/2/automated-vehic...

jacquesm 8y ago

Anybody still want to defend the police's claim within a day of the accident putting the blame on the woman that got killed?

hazardmat 8y ago
Yes. They said "preliminarily" the Uber driver was not at fault. Judging from the footage, that was reasonable. You can't put the blame off the woman, you can't just cross a road at night like that and expect to be noticed. Accidents like that happen all the time, it's only because of the self-driving car that this is newsworthy. Imagine if there wasn't any dashcam footage and this was a normal car with a distracted driver, without any other witnesses they would likely be off the hook.
- jacquesm 8y ago
  A distracted driver has the option to tell the police they were distracted. Or are you of the opinion that since the woman was dead it didn't matter anymore?
  - hazardmat 8y ago
    First of all, in this scenario, the woman would have at least shared some of the blame. Second of all, what are the odds that a human who just killed a woman would admit even to themselves that they were the one to blame in such an accident? Practically zero. Cognitive dissonance won't allow it. In their minds, they weren't distracted, at least not enough to take any of the blame.
    - jacquesm 8y ago
      > First of all, in this scenario, the woman would have at least shared some of the blame. Possibly. But a distracted driver is definitely at fault, no matter what. Driving is a full time occupation. > Second of all, what are the odds that a human who just killed a woman would admit even to themselves that they were the one to blame in such an accident? Pretty good. Most people are honest, even in a bad situation. > Practically zero. You speak for yourself. > Cognitive dissonance won't allow it. In their minds, they weren't distracted, at least not enough to take any of the blame. Well, you probably have your reasons for writing this under a novelty account. But it's clear that when you are at fault it is best for everybody involved to recognize that fact, even if it means discomfort for you. I get that in todays climate it's what you can get away with, but I don't think that strategy will hold in the longer term. It's way each and every lawsuit is appealed to the highest courts because nobody bothers to just admit their errors and it's why when there is a car accident the perpetrators will sue the victims. But that's wrong on many levels and I refuse to subscribe to that view.

w_t_payne 8y ago

I think that I am qualified to say that cost-effective testing is and will continue to be the barrier to self-driving car adoption. A big part of the solution to this problem is design-for-test; and the adoption and tight integration of a wide range of (quite conventional but sadly often overlooked) testing and systems engineering standards and processes.

toast0 8y ago

> We’re actively cooperating with the NTSB in their investigation. Out of respect for that process and the trust we’ve built with NTSB, we can’t comment on the specifics of the incident. Hey look, somebody knows how to follow the script! Uber may be grossly negligent, but at least they respect the process unlike Tesla.

TillE 8y ago
It's actually the rest of that statement which shocks me a bit: > In the meantime, we have initiated a top-to-bottom safety review of our self-driving vehicles program, So they actually think they can continue to work on self-driving vehicles after this, and not scrap the program entirely. That's...not what I expected. No municipality in the country should allow Uber to test on their streets after they killed someone, almost certainly because of their recklessly fast development. That's not something you come back from by promising to Do Better, especially not when there are more than a few competitors already way ahead of you.
- lhorie 8y ago
  > No municipality in the country should allow Uber to test on their streets after they killed someone This is pitchfork mentality. Many industries have to deal with unintentional deaths. Car manufacturers sometimes have to deal with deaths resulting from hardware failure. Food companies sometimes have to deal with recalls due to food poisoning deaths. If the response to a death is to ban all further production work, eventually nothing is ever going to get done because it's inherently impossible to be perfect all the time. It sucks that it had to come to someone's death, but the rest of the statement said: > we have brought on former NTSB Chair Christopher Hart to advise us on our overall safety culture so it sounds like Uber is at least trying in earnest to put the house in order.
  - cimmanom 8y ago
    Or a cynic could interpret that as "we've hired a crony of the people who will be judging us so he can lobby them to let us off easy".

SilasX 8y ago

Sorry for the snark, but there's a joke in here about: "Uber business development saw but ignored laws it broke."

Udik 8y ago
There is without doubt a grotesque irony in this story. Uber stands as a perfect symbol of the reckless capitalism of the 21st century: invests hundreds of millions in building a self-driving monster in the shape of a 2 ton luxury SUV, then puts behind its wheel- as a "safety driver"- a former convict for attempted robbery. And the capitalism-fueled metal golem immediately crushes a homeless woman pushing a humble bicycle. I mean, it could have been written as a satire of our times. Except, it really happened.
hughes 8y ago
Snark aside, false-positive rejection will be an aspect of any lawful autonomous vehicle. Incorrect false-positive rejection will be a reality, and will have to exist within lawful operation.

thegambit 8y ago

It seems like it was actually Uber who disabled the safety settings that were there to prevent this according to Volvo. "Uber Technologies Inc. disabled the standard collision-avoidance technology in the Volvo SUV that struck and killed a woman in Arizona last week, according to the auto-parts maker that supplied the vehicle’s radar and camera." https://www.bloomberg.com/news/articles/2018-03-26/uber-disabled-volvo-suv-s-standard-safety-system-before-fatality https://www.bloomberg.com/news/articles/2018-03-26/uber-disa...

nmca 8y ago
This whole line of reasoning frustrates me somewhat. The story of "Uber Disabled the Safety System" paints Uber out to be negligent for doing so. Outside of consideration of all the other facts (which unlike this one, do indeed suggest that Uber are to blame), this element of the story is not indicative of negligence. Of course they disabled the existing safety system - they were aiming to build a new, safer one that could operate without reliance on or interaction with the existing one. That (again, on its own) seems like a totally reasonable engineering call.
- linker3000 8y ago
  Seriously!? The responsible position would be to retain the existing system in a passive standby mode, capable of overriding the active 'under test' one in the event certain safety/detection thresholds were exceeded.
  - lhorie 8y ago
    IIRC someone had said that there are physical limitations to what one can actually do (namely, having to physically unplug the connector for one safety system in order to be able to add your own at all)
    - telchar 8y ago
      Sure, if you want to move fast and disrupt yada yada. They shouldn't be putting hack-job self-driving cars on public roads. Either do some engineering so that they keep the backup system in place or keep it on a test track. This is Chernobyl-like irresponsibility.

sharemywin 8y ago

Is this a binary decision? I normally slow down when I see a something at the side of the road.

wilburTheDog 8y ago

If I understand it right, this is the biggest blind spot of all the automated driving systems. The software just ignores stationary objects because the sensors don't have the resolution to determine whether they are in front of the car or on the side of the road. And if it didn't ignore stationary objects the car would come to a halt at every garbage can, mailbox, and bush along the road. Until they can work lidar into the system this will not be preventable. And, correct me if I'm wrong, but I think the vast majority of accidents that were the fault of the self-driving car happened because it hit something stationary in the road (or off to the side of the road, in one case I recall). Since I've learned about this blind spot in these system I no longer trust them. I think everyone with a self-driving car should understand this. But maybe it's not in the interest of the manufacturer to explain it, or people would demand they fix the problem, which they cannot do just yet.

Symmetry 8y ago
That wasn't the case at all for this collision. As the article says, the sensors had detected the woman but the information had been disregarded. And the moving object thing is exclusive to radar, vision systems don't suffer from it though vision systems have their own problems.
- wilburTheDog 8y ago
  The article linked mentioned "B was the problem", meaning it ignored the sensor input. Another article(1) about it mentions that Uber had reduced the number of LIDAR sensors from five to just one rooftop sensor. The president of the company that builds the sensors even said you would need side sensors to see pedestrians, especially at night. But LIDAR is expensive. So it seems that, like Tesla, they were trying to rely on radar for that kind of thing. But that does leave the blind spot(2) I mentioned. Given that I don't see how this case is so different. Though I still don't understand how, specifically, that blind spot problem works. They say when the car in front of you changes lanes and a stopped truck is now in your lane you'll hit it. What if no car changed lanes in front of you? Would you still hit a stopped truck in your lane? The guy whose Tesla hit the bridge column seems to tell me yes, but I'm not sure. 1. https://www.theregister.co.uk/2018/03/28/uber_selfdriving_death_may_have_been_due_to_lidar_blind_spot/ https://www.theregister.co.uk/2018/03/28/uber_selfdriving_de... 2. https://www.wired.com/story/tesla-autopilot-why-crash-radar/ https://www.wired.com/story/tesla-autopilot-why-crash-radar/

Animats 8y ago

I'm looking forward to seeing the full NTSB report. This hints that Uber's mindset is focused on "obstacles", not "flat road". The first job of automated driving is to drive only on flat surfaces. Doesn't matter why it's not flat. Then you worry about where to go. That's how the off-road DARPA Grand Challenge vehicles had to work. "Flat road" is a pure geometry thing. Not much AI needed. On road, non-flat road is rare, and you don't have to worry much about going off cliffs, rocks in the road, and such. So it's tempting to focus on "obstacles" to be tallied and classified. Tesla definitely has an "obstacle" focus, and a limited class of obstacles considered. Waymo profiles the ground. Looks like Uber had the "obstacle" focus. If you have a "flat road" detection system, things the obstacle detector doesn't understand get stopped for or bypassed. So the vehicle isn't vulnerable to this flaw. There's a higher false alarm rate, though. And a non false alarm problem. A piece of trash on the road is likely to result in a slowdown and careful avoidance, not a bump as the car rolls over it. Still, better to take the safe approach unless you're on a freeway and the cars ahead of you just got past the obstacle successfully. The Udacity self driving car "nanodegree" trains people to build obstacle detectors and classifiers. That may be a problem.

krschultz 8y ago
Is another way of framing this: "flat road" => assume you can't continue until you prove it's safe "obstacle detection" => assume you can continue unless there is something blocking the current path The bit of autonomous vehicle work that I did started from a set of potential trajectories that were then pruned by obstacle detection. So we always assumed you could continue moving forward unless every potential trajectory was pruned. We ignored the cliff problem as beyond our scope.
dotancohen 8y ago
> The first job of automated driving is to drive only on flat surfaces. Doesn't matter why it's not flat. Then you worry about where to go. The old adage applies. Aviate, Navigate, Communicate. In that order.

jessaustin 8y ago

ISTM that HN policy would replace TFA with the link cited in TFA: https://www.theinformation.com/articles/uber-finds-deadly-accident-likely-caused-by-software-set-to-ignore-objects-on-road https://www.theinformation.com/articles/uber-finds-deadly-ac... ...although, there is a paywall, so maybe not.

jessriedel 8y ago
Not just any paywall either. It's $40/month.

deleted 8y ago

[deleted]

mcr1983 8y ago

Uber's ultimately at fault because it was their car. But the woman driving should be arrested for vehicular manslaughter. Uber knew its cars weren't autonomous, that's why they had a driver to take over when the car screws up. But she was texting while she should've been ready to take control of the vehicle and hit a pedestrian any driver that was paying attention would've easily avoided.

bkor 8y ago
> Uber's ultimately at fault because it was their car. Uber while hiring should've specifically looked for people who can pay attention for hours on end. Basically like a train driver. Further, they should review if people pay attention and have a system in place. She was employed by Uber. If you make a huge error while working for someone, often it's the employer who is responsible. There's way too much which should've been done which wasn't to just blame this driver.

bsaul 8y ago

i haven't read much about the observation that driving = communicating. aka : whenever you drive, you send and receive information from all the environment and human, establishing sometimes what could be similar to a dialogue. Doesn't this implie that autonomous driving won't be solved until we're able to have "true" personnal assistant ? The message in driving are simpler than with a regular voice communication, but the objects and concepts visually present seem just as diverse.

chiefalchemist 8y ago

"Saw" doesn't seem like the right word. Granted, apparently, the sensors picked something up. What it failed to do is recognize and categorize; and from there properly take action. Given the failure of the recognition + categorization system, it would be interesting to know what it thought it saw. Was it filed under IDK, a plastic bag blowing in the wind, what?

someonewhocar3s 8y ago

One of the final tests, mandatory for regulatory acceptance: Produce accident on target individual within timeframe

deleted 8y ago

[deleted]

m3kw9 8y ago

What’s worse, it saw it but misinterperated or sensors has a blind spot and missed something?

stillsut 8y ago

The exact circumstances of the struck pedestrian likely violated many [possibly curde] bayesian priors for what to consider a positive collision threat: A rogue pedestrian at an unflagged portion of road, at an odd time of night, on a road that doesn't usually have pedestrians, with a bike moving perpendicular across the road (instead of parallel along it). Each one of these had been learned to be a high false positive to true negative ratio. The product of these rarely encountered events (even independently) allowed a higher level algorithm to "score" the approaching unverified object through bayesian inference, below the level of a reasonable expectation of human collision. In a way, this system should never be expected to become overly reckless or feckless as compared with even top human drivers in the long run: close calls of this nature should be input to the system (perhaps deliberately through staged QA?) and added to the model of collision threat identification.

dmix 8y ago
That is the classic self-driving tradeoffs question, no? I believe rather than absolutes, the solution is her is properly identifying a serious crash causing injury was missed. Regardless if it was ID'd as human, it was still someone pushing a shopping cart full of bags across a highway intersection that could have been detected. Even if it wasn't human, and totally dangerous and improper crossing at night time on a busy highway, it should have stopped even for the drivers own safety.
justicezyx 8y ago
I am rather confused: Should the car always respond by slow down as the mandatary response when there is object in front? Are you implying that the algorithm would conclude that because it's unlikely that there is an object in that circumstance, it is ok to disapprove its own observation that there is some thing in front of the car? That sounds utterly nonsensical.
- danenania 8y ago
  Exactly. I'd expect collision-prevention to have absolute priority over any other system. It should not be possible for any logic bug or combination of environmental factors to make the car run into a large object in its field of vision. There's no need for statistics here--an overriding 'don't run into stuff' rule will suffice. If there's truly no possible route to avoid some collision and therefore the car needs to decide which collision is preferred, then the statistics can kick in, but in this Uber scenario, it shouldn't have gotten that far. From the outside, it seems like a fundamental design issue.

jiveturkey 8y ago

i wonder how this info got out. i thought (learned this from tesla) that companies are not allowed to release info about accidents under NTSB investigation. which this one is.

dingo_bat 8y ago

> This puts the fault squarely on Uber’s doorstep, though there was never much reason to think it belonged anywhere else. > This is not good. The state of journalism saddens me.

erdo 8y ago

I'm wondering how much more attentive the emergency driver would have been if they were one of the developers. Sometimes I worry about how much trust non-developers assign to software (in all sorts of areas)

kalal 8y ago

Sounds like the departments are fighting over the responsibility of the accident. Correct me if I am wrong.

jlebrech 8y ago

Ethical dilemma, should a vehicle swerve to avoid an unavoidable pedestrian and potentially killing more people or should it ignore that person (not saying that what happened) Should a autonomous vehicle also prioritise it's own passenger over the safety of others, aka head on collision with a people carrier.

agumonkey 8y ago

it's dramatic but it's also a good reminder that "modern" "recent" wonder tech doesn't comes free. It needs proper examination. This will surely push toward more serious work.

dboreham 8y ago

Um...if I as a human see a plastic bag on the road ahead, I slow down to be able to carefully ascertain if it perhaps is a plastic bag full of nails. I don't drive full tilt over it.