31 ms·

Uber’s self-driving car could not detect pedestrians outside of a crosswalk

• tekknik 7y ago
  I still fail to see what benefit self driving cars give us. There’s always going to be bugs and now we’re strapping them to multi-tom vehicles. This reeks of bad idea yet we continue on as if this is the a cure for some huge problem
• gourou 7y ago
  People jaywalk often, I feel like this issue should have appeared before. Would it be possible Uber was using some of Otto's technology (self-driving trucks) then decided to replace it abruptly (because of the lawsuit with Google) and it caused this seemingly avoidable crash? Context: Uber acquired Otto, a company founded by Google Waymo's former CEO, Anthony Levandowski. It quickly got involved in a lawsuit where Google alleged that Lewandowski stole Waymo's self-driving intellectual property. Uber later agreed not to use any Waymo IP and give 0.34% of equity to Google. https://www.buzzfeednews.com/article/priya/waymo-asks-judge-to-halt-ubers-self-driving https://www.buzzfeednews.com/article/priya/waymo-asks-judge-... https://jalopnik.com/googles-waymo-and-uber-reach-settlement-in-high-profile-1822868808 https://jalopnik.com/googles-waymo-and-uber-reach-settlement...
• ForHackernews 7y ago
  Between this and deliberately disabling Volvo's own emergency braking features, I think some decision-makers at Uber should be charged with negligent homicide. Maybe a few executives going to jail will put an end to this cowboy attitude where lives are on the line.
  • latentpot 7y ago
    Jail in corporate America? Sacrilege!
  • microtherion 7y ago
    IANAL, but given that the disabling of the security system was an act of commission, not omission, it could be argued that this was more than just "negligent". On the other hand, car manufacturers have no legal duty to include collision avoidance systems, so disabling one is probably not illegal in itself.
• dghughes 7y ago
  And now today (Nov 6) there are news articles about the Tesla "Smart Summon" which was quietly activated. There are videos of dangerous near collisions of "summoned" driver-less Tesla vehicles very slowly driving among other vehicles. https://www.ctvnews.ca/video?playlistId=1.4658613 https://www.ctvnews.ca/video?playlistId=1.4658613
  • tdy_err 7y ago
    That’s very blatant sensationalist journalism that conveniently does not mention that there is a person holding down a button the entire time that the car is being summoned. If that’s “dangerous”, well, so is piloting it manually.
    • mijamo 7y ago
      A person who does always what is in front of the car and what is coming behind or on the sides, and who will probably be looking at the button rather than the car while it moves. That is just as dangerous as using your smartphone while driving in a parking lot, which is forbidden for good reasons.
      • magashna 7y ago
        Do you stare at a button when you hold it? That seems very odd. If that were true, video games wouldn't exist.
        • ceejayoz 7y ago
          The video shows the app with a radar-ish view of what the car is seeing of its surroundings and its planned route. It's more than just a button, and I could see why someone would be focused on the screen display instead of the car.
    • boomlinde 7y ago
      I have a similar sentiment about nuclear weapons. What's the problem if someone needs to be pushing a button?
• isostatic 7y ago
  It wasn't that the sensors were confused or something - the AI had no idea how to cope with pedestrians in the road for whatever reason, so threw them in the "Other" category. At least 98% of objects are categorised, the rest don't matter. /s
  • rvense 7y ago
    TBH this does sound exactly like something you'd expect to happen if you took some Javascript ninja used to moving fast and breaking things and asked them to build a car. But surely Uber has more sense than that...
    • svrtknst 7y ago
      Yes... surely.
    • dsfyu404ed 7y ago
      Coming from the defense (go ahead, clutch your pearls, I won't be offended) industry I just can't understand why people ship things that fail into in potentially dangerous states on some known non-trivial fraction of inputs. If the purpose of the software were entertainment or some other low stakes thing I'd understand but software controlling things that can kill people or lose lots of money need to be able to handle unexpected inputs gracefully.
      • Eric_WVGG 7y ago
        Because the penalties and rewards don’t align.
      • Majromax 7y ago
        > I just can't understand why people ship things that fail into in potentially dangerous states on some known non-trivial fraction of inputs Because that kind of safety-conscious culture needs to be deliberately enforced, from the top all the way down, by a system that takes a long view. If we're making an analogy to the defense industry, then building self-driving cars as a tech startup is closer to the Anarchist's Cookbook than Northrop Grumman.
• reedwolf 7y ago
  To make an omelette, you have to break a few eggs.
  • rq1 7y ago
    Do you want to be the egg? Or your husband/wife? Maybe your children?
  • me_me_me 7y ago
    Well I hope you are going to be that egg not me. Cuz multi billion corp is too cheap to properly test their product and just deploy into production and see what happens and patch it later.
    • reedwolf 7y ago
      We can either have fast technological progress, or we can be completely tied up in bureaucratic red tape, with every potential innovation having to go through endless "oversight" committees staffed by non-technical airheads. I know which I prefer.
      • kranner 7y ago
        > We can either have fast technological progress, or we can be completely tied up in bureaucratic red tape, with every potential innovation having to go through endless "oversight" committees staffed by non-technical airheads. https://en.wikipedia.org/wiki/False_dilemma https://en.wikipedia.org/wiki/False_dilemma
      • FroshKiller 7y ago
        That's a false dilemma.
      • whatisthiseven 7y ago
        That is a false-dichotomy. We can have something more in the middle, or at least less an extreme as "let cars just drive wherever without any tests and when it kills someone just say o well, that's progress". In this particular case, Uber got a deal with the mayor (Governor? I can't remember) of the city to do testing, and it was clear the government bureaucrat allowed them to do whatever, and then coordinated with the police after Uber killed a pedestrian to fabricate evidence that looked like the woman came out of nowhere. I would say that is really far from "endless oversight" and much more into the "just mow them down one at a time until you get it right" territory.
• jefftk 7y ago
  The NTSB just released their report: https://dms.ntsb.gov/public/62500-62999/62978/629713.pdf https://dms.ntsb.gov/public/62500-62999/62978/629713.pdf "If the collision cannot be avoided with the application of the maximum allowed braking, the system is designed to provide an auditory warning to the vehicle operator while simultaneously initiating gradual vehicle slowdown. In such circumstance, ADS would not apply the maximum braking to only mitigate the collision." "Certain object classifications— other—are not assigned goals. For such objects, their currently detected location is viewed as a static location; unless that location is directly on the path of the automated vehicle, that object is not considered as a possible obstacle. Additionally, pedestrians outside a vicinity of a crosswalk are also not assigned an explicit goal. However, they may be predicted a trajectory based on the observed velocities, when continually detected as a pedestrian."
  • DougBTX 7y ago
    > continually detected as a pedestrian Interesting, from the article and this, it sounds like the system can’t maintain position tracking of an object if it’s classification changes. So even if it could detect a pedestrian, something that could be ambiguous like a pedestrian pushing a bike might have no motion tracking data from one moment to the next, so the car would have no ability to predict its trajectory.
    • tjungblut 7y ago
      I had the same impression from reading the table. Looking at the map and trajectories though, how could the human not see the car coming? Or was she thinking that the car would stop/slow down? Same question holds for the vehicle operator. edit after reading the other reports: the victim apparently was under the influence of methamphetamine and the vehicle operator was busy watching Hulu.
      • DougBTX 7y ago
        As far as I recall the driver was looking at the car's status screen which they'd been instructed by Uber to monitor rather than at the road at the time of the crash, so didn't notice the pedestrian until too late. I can't say much about what the pedestrian saw, the Uber car killed them so we can't ask.
• CommieDetector 7y ago
  No the driver (human) killed that woman.
• fabian2k 7y ago
  With a disabled emergency braking system and an inability to handle a rather common situations like people jaywalking, these cars really shouldn't have been driving in public. That is the kind of stuff that I'd expect to be tested and implemented on private grounds. Additionally, reducing the number of people in the car to one when the car is pretty much by design not capable of handling emergency situations by itself is quite reckless.
  • dmix 7y ago
    They shouldn't have been driving in public with someone who wasn't paying attention 100% of the time as any regular driver would. The longer quote: > Also, don't forget: the SUV's emergency braking system was deliberately disabled because when it was switched on, the vehicle would act erratically, according to Uber. The software biz previously said “the vehicle operator is relied on to intervene and take action," in an emergency. The question then was there proper training and communication to the test drivers that it's never okay to look down at your phone. Or whether that was simply unrealistic expectations. Or if the hours were too long, or testing at night, etc. It said there was 5 seconds which should have been more than enough for a human test driver to hit the brakes, which was their stated job.
    • madamelic 7y ago
      >The question then was there proper training and communication to the test drivers that it's never okay to look down at your phone. If I remember right, Uber did cost-savings by removing the safety engineer and expected the driver to fill both roles simultaneously.
    • Symmetry 7y ago
      In this case the safety drive was watching their phone which was playing a movie so a bit beyond the normal level of attention wandering you'd expect. But even for people trying their best having to pay attention to a road for hours without having to provide any input isn't something you can reasonably expect people to do. NHTSA level 3 autonomy is just a bad idea, we need to go straight from 2 to 4.
• wongarsu 7y ago
  So uber fails to predict even completely linear paths traveled by objects if the system isn't trained on or expecting that specific type of object? That sounds like an even bigger issue than "we didn't think pedestrians could exist outside designated crossings"
  • madamelic 7y ago
    What's crazy to me is that an "Other" object isn't flagged for a complete stop or a major alert to the "safety" driver. This seems like a major case for when the software should stop being autonomous, when it isn't sure. Likely the story is that their software flags a lot of things "Other" without figuring it out upon closer inspection.
    • pluma 7y ago
      For those who think "well, pedestrians don't belong on the road so this wasn't much of a bug" (because I'm sure there are some of those): just imagine the "other" was a large chunk of debris, or cargo fallen off a truck. The car would have crashed right into it. There can be many obstacles on the road, stationary or non-stationary. AI can get away with requiring humans to take control in emergency/unusual situations, but not when it is supposed to be "self-driving". The software at least needs to recognise problematic situations and request a safety override or come to a safe stop, it can't just shrug it off and move on.
      • Faark 7y ago
        Debris tends to be stationary most of the time. Sounds like the car could have handled that. I agree with that ADS being shit. Objects, once identified, should be accounted for in later updates in some form. E.g. in the way of "pedestrian now behind truck next to me". How the hell can an object switch status back and forth, loosing all its history? This should make the system be extra cautious at the very least. But yeah, safety doesn't seem to have been that big of a priority.
        • pluma 7y ago
• deleted 7y ago
  [deleted]
• Simulacra 7y ago
  If the safety driver had been paying attention to the road instead of his phone, they might have been able to break in time so that this woman did not get hit.
  • Filligree 7y ago
    Human minds don't work that way. Paying continuous attention to a vehicle you're not controlling is quite simply impossible, and if safety relies on it then your safety system is broken.
    • jimmaswell 7y ago
      Human minds don't work that way. Paying continuous attention to an airplane you're not controlling is quite simply impossible, and if the function of a copilot relies on it then airlines are broken. People are tasked with monitoring things all the time. It's not some brand new impossible task.
      • Filligree 7y ago
        There's a critical difference between airplanes and cars. When flying an airplane, very few emergencies ever require action faster than "Couple of seconds" -- and any procedure which does require such will be thrown out at first opportunity.
  • MereInterest 7y ago
    Possible, but should not be relied upon. Even in cases where the driver knows that a handoff to manual control will occur, it takes a significant amount of time to mentally switch from being a passenger to being a driver. For cases where the AI knows it isn't performing well, such as heavy rain or snow, it makes sense to handoff. For anything that happens on shorter than a 10 second timeframe, the AI must be able to handle it, because the human cannot switch modes that quickly.
  • KeithBrink 7y ago
    The interview with the driver by the NTSB is interesting if you want more details from the driver's side of the story: https://dms.ntsb.gov/pubdms/search/document.cfm?docID=477745&docketID=62978&mkey=96894 https://dms.ntsb.gov/pubdms/search/document.cfm?docID=477745... You can also see the detailed report from the NTSB which will go through about everything you would want to know about the actual driver and her reaction: https://dms.ntsb.gov/pubdms/search/document.cfm?docID=477743&docketID=62978&mkey=96894 https://dms.ntsb.gov/pubdms/search/document.cfm?docID=477743... Particularly of note is that Hulu was streaming video on her phone until about 1 minute after the crash (page 11 of the report).
• michalc 7y ago
  As a UK-resident, I find the omission of considering a pedestrian in the road quite unexpectedly, and unfortunately, overly car-centric. There isn’t really the term “jaywalking” here. It’s just “crossing the road”. I’m not sure on exactly who has what legal responsibility, but it certainly feels like pedestrians should look out for cars when crossing, and drivers should look out for pedestrians.
  • deleted 7y ago
    [deleted]
  • Razengan 7y ago
    [nevermind, the case I linked wasn't a pedestrian]
    • Symbiote 7y ago
      That seems thoroughly off topic.
    • w457uiw4gftyi 7y ago
      That case involves neither autonomous cars nor pedestrians.
      • Razengan 7y ago
        My bad, people commented too quickly for me to delete it when I realized it.
    • fortran77 7y ago
      It was a pedestrian, _walking_ a bicycle.
      • Razengan 7y ago
        I posted about Harry Dunn who died while "riding his motorcycle when a woman emerged from the airbase on the wrong side of the road and there was a head-on collision." I mistakenly recalled that case as involving a pedestrian.
• jon889 7y ago
  What does it matter what the "object" was, if anything/something is moving to intercept then the car should stop.
  • pbhjpbhj 7y ago
    Well surely not, if its a bird and you'd cause an accident with an emergency stop then you should continue. Dogmatic responses are exactly the problem with self-driving vehicles. Sometimes it's preferable to hit something in your own vehicle. Cue the classic moral question of which person would you kill if you can only avoid one - with your vehicle - by hitting another.
    • Symmetry 7y ago
      Generally we should probably have vehicles follow proscriptions of the law when they can't avoid hitting anything and change the law if we want them to act differently. I've never seen a hypothetical self-driving car trolley problem where there wasn't a single option that was clearly what the law required.
      • ghaff 7y ago
        By and large, the trolley problem concept is overblown. Most of the time the right/best answer is going to be to stand on the brakes and hope for the best. But I'm honestly not sure what the law "requires" if you've got a scenario where there are going to be bad outcomes no matter what you do.
        • Symmetry 7y ago
          For instance if a car has the choice of hitting someone in the road or swerving onto the sidewalk and hitting someone there then clearly the legal thing to do is for the car to stay in its right of way and hit the person in the road.
          • ghaff 7y ago
• vanon 7y ago
  If you think Uber’s car is bad can you imagine how bad Lyft’s is or most any company that started years behind? I worked in the SDC industry. It’s mostly a science project. Between its economics and its car I don’t understand why Lyft isn’t heavily shorted.
  • dmix 7y ago
    > If you think Uber’s car is bad can you imagine how bad Lyft’s is or most any company that started years behind? The amount of time someone spent on a software project isn't the best indicator of its quality... especially for science work where there's a large amount of knowledge transfer within the field. Lyft hired a top Google engineer to run the project and plenty of other experienced people. They weren't starting from scratch like Waymo and nor did Uber. The car simply shouldn't have been tested without a driver (or two) constantly paying attention. Clearly Uber didn't trust it to be running by itself yet (especially with the emergency brakes being disabled) but it basically was if the test driver wasn't paying attention.
• fredley 7y ago
  Jaywalking was invented by the car industry, I wonder if they'll double down on it again: https://www.vox.com/2015/1/15/7551873/jaywalking-history https://www.vox.com/2015/1/15/7551873/jaywalking-history
  • wongarsu 7y ago
    Maybe works for Uber and GM, but in Europe jaywalking doesn't exist. If they want to sell cars here they will have to deal with being responsible for almost every pedestrian they hit, because that's the standard we already set for human drivers.
    • skrause 7y ago
      > but in Europe jaywalking doesn't exist Europe is not a country and each European country has different laws: https://en.wikipedia.org/wiki/Jaywalking#Europe https://en.wikipedia.org/wiki/Jaywalking#Europe
      • wongarsu 7y ago
        But it is a somewhat homogeneous region with similar laws, as supported by that Wikipedia article: in general you can cross roads anywhere unless it's a motorway or you are within 100 meters of a designated crossing (any closer and the laws diverge on what's acceptable). That's just how we talk about the US treating things a certain way despite the different states having varying laws with countless exceptions and special cases.
        • Piskvorrr 7y ago
          In which case, this would have been a jaywalking infraction in many European countries, as a designated crossing is nearby. However, it is not the role of a driver (or their car) to serve death penalties for that.
          • josefx 7y ago
• munmaek 7y ago
  What’s that quote about the last 20% of the work taking 80% of the effort? This crash seems to be the result of several cases not being handled correctly, or at all. 1. Pedestrian crossing -not- at a crosswalk, aka jaywalking 2. Objects tagged as “Unknown” don’t have their path tracked. By the time the car saw an “unknown object” (the woman) directly in its trajectory it was too late (1.2s before impact). Why was there no system for a person jaywalking? That’s extremely common. Why don’t “Unknown” objects get stored and individually have their trajectories tracked? Hello? Programmers need to start going to jail.
  • pluma 7y ago
    Not necessarily the programmers. There'll always be some people who are easily enough to manipulate to write this kind of code. But those making the decision to ship this and sign off on it definitely should face criminal negligence charges.
    • munmaek 7y ago
      One can only claim ignorance for so long. They know what they’re programming. But yes, managers and the like involved should also be going to jail. This cavalier, negligent attitude needs to be dealt with yesterday.
  • brenden2 7y ago
    Pareto principle: https://en.wikipedia.org/wiki/Pareto_principle https://en.wikipedia.org/wiki/Pareto_principle
  • Sohcahtoa82 7y ago
    > Why don’t “Unknown” objects get stored and individually have their trajectories tracked? Hello? Because what if that "unknown" object is just a plastic bag blowing in the wind? People would get mighty annoyed very quickly if emergency braking kicked in for every piece of garbage blowing onto the road.
    • TallGuyShort 7y ago
• rom1v 7y ago
  For reference, the video recorded by the car: https://twitter.com/TempePolice/status/976585098542833664/video/1 https://twitter.com/TempePolice/status/976585098542833664/vi...
  • stefan_ 7y ago
    No, this is a video from a shitty dash cam released by Uber (and then happily compressed to shit and forwarded by police, who don't miss a chance to blame whoever died in a crash) to deliberately misrepresent the situation. This garbage pinhole sensor has not one hundredth of the dynamic range of the actual cameras they use for the autonomous driving and is not at all representative of how human eyes see, which would have had no problem spotting her from many many seconds away. No, this is part of the crime here.
• mattacular 7y ago
  Here's a simple rule that could be enacted for companies seeking to do work in the autonomous driving space: if your car kills a person, your company loses its license to work on autonomous vehicles. Forever. That will dictate the adequate pace to achieve these goals safely. Then we'll see what's really possible with this technology. Having cars that can drive themselves just doesn't seem like a particularly high priority for society at large in the face of other looming issues. Why allow it to proceed in such a dangerous fashion at all?
  • drcode 7y ago
    What if the rate of deaths for a given company is not zero, but below the rate for human drivers? Is it ok to have additional, unnecessary pedestrian deaths by NOT allowing that company to deploy their technology? (That said, the negligence in the Uber case makes it pretty clear they are likely far from reaching that level of competency)
    • puranjay 7y ago
      So you mean being okay with a multibillion dollar corporation harming people so it can make even more profits, all the while telling you that it's "good for society" Gee, where have I heard that script before?
      • nulld3v 7y ago
        So is reducing the accident rate on roads "harming people"?
        • puranjay 7y ago
          Good for the people once the system is perfected. Not so much for the ones who get run while the system is being perfected. The "greater good" utilitarian argument has been the basis of some of the worst policies and politics in the world. I'm not saying that self driving cars fall into the same category, but how many deaths are you okay with until Uber/Waymo perfect their algorithms (and later, charge you for it)? 1? 10? 100?
• fortran77 7y ago
  Some blame, too, belongs to the human driver. His job was to look for exceptional conditions that the car may not pick up. And he failed. It's quite possible there wasn't time for a good human driver to stop completely or avoid a pedestrian where one shouldn't be, and this accident was all but unavoidable, but we'll never know because no attempt was made. Given the speed of the car, the woman crossing the street either miscalculated, was betting that traffic would see her and slow down, or didn't see the oncoming car. It was also odd that they disabled the car's stock emergency collision avoidance system.
• chagai95 7y ago
  this definitely counts as a human error, so stupid.
• skc 7y ago
  Self driving cars probably (and impossibly) need their own roadways like cable cars or trains.
  • madamelic 7y ago
    So... trains?
  • javagram 7y ago
    So, self-driving BRT? Probably will only happen when self-driving cars do in general.
• hirundo 7y ago
  When the software I write fails, and it does, someone doesn't get to read a web page. Nobody dies or even gets very inconvenienced. This story reminds me of how good a thing that is. After 40 years of practice I'm still not competent to write such high stakes code. I hope the people who do write it are far more competent than me, but still at least as skeptical about their own capability.
  • AgloeDreams 7y ago
    An interesting trap is people in the field of social media (FB/ Twitter) not realizing that they are in that high-stakes environment where bad privacy settings and leaks can cause death.
    • fredley 7y ago
      I mean with regards to Facebook in particular, bad settings can cause genocide. https://www.bbc.co.uk/news/blogs-trending-45449938 https://www.bbc.co.uk/news/blogs-trending-45449938
    • twitter_anon 7y ago
      Some of us do, as we come from (or work with people who come from) states that will perpetrate violence against dissidents. Unfortunately, there still aren't enough of us. We're trying to change that.
  • massysett 7y ago
    (I don't know how competent you are, but it's likely that) nobody is more competent than you are. We solve high-stakes situations not by installing different people, but by installing better systems and by double-checking things. In my job we ensure the quality of high-risk things by having multiple layers of review and quality control. It doesn't matter how good the people are who are assigned to the task. We need double-checks. Lower-risk things are not subject to this level of quality control. Most computer software is low-risk and so it gets very little review. Unfortunately some computer software is much higher risk but does not seem to get review commensurate with its risk.
• taylodl 7y ago
  When I learned to drive they stressed unanticipated objects coming into the street - especially balls: don't look at the trajectory of the ball and judge whether it will be in your way, instead look for the kid who will be running after the ball while not paying attention to traffic. Also in my state if you hit a kid (a minor actually) on a bicycle you are guilty until proven innocent. That's the law. You have to prove there was no reasonable way for you to avoid the accident.
• deleted 7y ago
  [deleted]
• patagurbon 7y ago
  It’s mind boggling that tracking history wasn’t maintained between object classifications. Surely the best thing to do would be to detect "object" -> track it -> classify -> predict path -> back to classify
• hexagone 7y ago
  I almost jaywalked while reading this. Thankfully the vehicles are all manually driven.
• mangecoeur 7y ago
  So: Uber put a car on the road that was not designed to detect pedestrians except in a few limited conditions. Someone died because of their gung-ho approach to safety. How is that even remotely acceptable? Really hope we see people being held responsible for this...
• manbearpiggy 7y ago
  This article is just terrible, it goes into the code/technical problems with the car. They killed a woman and no one went to jail for manslaughter.
• hysan 7y ago
  Animals crossing the street is rare, but a situation I'd expect self-driving cars to be trained on. I doubt they classified all known large animals, so shouldn't there have been a classification to anticipate animals it doesn't know crossing the street? As someone who spent a number of years living in the countryside, this seems obvious. However, for those growing up in cities, this might never cross your mind. It makes me wonder if there might be a lack of diversity on the teams building and testing these systems.
• yumraj 7y ago
  We had a thread a few days back with optimistic people predicting self driving cars in 2020. I will urge those people to read this and consider the numerous other cases, edge or not, that these systems which seem to work great in the surface, may not be handling.
• WhompingWindows 7y ago
  "The most glaring mistakes were software-related. Uber’s system was not equipped to identify or deal with pedestrians walking outside of a crosswalk. Uber engineers also appear to have been so worried about false alarms that they built in an automated one-second delay between a crash detection and action. And Uber chose to turn off a built-in Volvo braking system that the automaker later concluded might have dramatically reduced the speed at which the car hit Herzberg, or perhaps avoided the collision altogether." Wow, dangerously and maliciously negligent engineering. Another reason Uber is a company to be wary and critical of. It's hard to say whether their rushed and corner-cutting approach has set back SDCs, but those engineers should change their ways now that a death is on their conscience.
  • bdcravens 7y ago
    > a death is on their conscience How much negligence is required before a death is on their criminal record?
    • me_me_me 7y ago
      Ask Boeing about that.
      • krapht 7y ago
        In this case, isn't the blame on Vasquez, the safety driver? If a plane crashes while on autopilot, ultimately we should blame the human pilot since they retain master control.
        • stlHusker 7y ago
          Given the nature of the system, when would she have been notified that she needed to take control? Additionally, what did Uber tell her about how the system operated? Ultimately it is on Uber for the lack of serious consideration. Rolling out level 2 & 3 autonomous driving systems where driver attention (especially non-experts in an open world) is still required on a wide scale is absolutely dangerous given human nature. Most people believe that you have to completely jump to level 4 & 5 autonomy which represents fully autonomous systems. Think of it like this: "Hey, let's get this person who isn't an employee of the company, gets paid a pittance in a "gig" economy where extreme "hustle" is needed, probably isn't an attentive driver anyways and put them as a critical backstop in a system that is still a prototype and that they most likely don't understand." The Uber engineering team lacked or ignored human factors experts.
• Symbiote 7y ago
  Why does Uber use an SUV? These vehicles are widely known to be much worse for pedestrian injuries and fatalities. They should use an ordinary car. (Frankly, I'd like to see all SUV drivers considered legally negligent in any pedestrian accident due to their choice of vehicle.)
• brudgers 7y ago
  The car hit Elaine Herzberg. Elaine's Obituary Elaine Marie Wood-Herzberg, 49, of Tempe, AZ passed away on march 18, 2018. A graveside service took place on Saturday, April 21, 2018 at 2:00pm at Resthaven/Carr- Tenney Memorial Gardens in Phoenix. Elaine was born on August 2, 1969 in Phoenix, AZ to Danny Wood and Sharon Daly. Elaine grew up in the Mesa/Apache Junction area and attended Apache Junction High School. Elaine was married to Mike Herzberg until he passed away. She was very creative and enjoyed drawing, coloring, and writing poems. She always had a smile on her face for everyone she met and loved to make people laugh. She would do anything she could to help you, and was there to listen when you needed it. Elaine is survived by her mother Sharon, of Apache Junction, AZ; her father Danny, of Mayor, AZ; her son Cody, of Apache Junction, AZ; her daughter Christine, of Tempe, AZ; her grandchildren Charlie, Adrian, and Madelyn; her sister Sheila, of Apache Junction, AZ; and many other relatives. But a homeless person is only ten points. So let's remember the car. https://www.sonoranskiesmortuaryaz.com/obituary/6216521 https://www.sonoranskiesmortuaryaz.com/obituary/6216521 https://afscarizona.org/2018/05/14/the-untold-story-of-elaine-herzberg/ https://afscarizona.org/2018/05/14/the-untold-story-of-elain...
  • sschueller 7y ago
    This is why I will never use any service from Uber ever. I may be insignificant but at least I know I am doing my little part to not support this disgusting movement of "Move Fast and Break Things" which includes actual casualties. Everyone involved except the lowest pawns get away with it so the next company can do it even worse.
    • solean 7y ago
      I hope you don't ever set foot in any other car either, considering they kill a million people every year. But a million is just a big number, it's not as powerful as 1 obituary.
      • perl4ever 7y ago
        This accident didn't save any lives though. It's just associated in your mind with saving lives. It's one more death; there's no decrease in traffic accidents to compensate.
• Strilanc 7y ago
  > 5.6 seconds before impact, it classified her as a vehicle. Then it changed its mind to “other,” then to vehicle again, back to “other,” then to bicycle, then to “other” again, and finally back to bicycle. System can't decide what's happening. > It wasn’t until 1.2 seconds before the impact that the system recognized that the SUV was going to hit Herzberg System is too slow to realize something serious is happening. > That triggered what Uber called “action suppression,” in which the system held off braking for one second A hardcoded 1 second delay during a potential emergency situation. Horrifying. I bet they added it because the system kept randomly thinking something serious was going to happen for a few milliseconds when everything was going fine. If you ever find yourself doing that for a safety critical piece of software, you should stop and reconsider what you are doing. This is a hacky patch over a serious underlying classification issue. You need to fix the underlying problem, not hackily patch over it. How is this not the title of the story? This is so much worse than the "it couldn't see her as a person, only as a bicycle". At least the car would still try to avoid a bicycle, in principle, instead of blindly gliding into it while hoping for the best. > with 0.2 seconds left before impact, the car sounded an audio alarm, and Vasquez took the steering wheel, disengaging the autonomous system. Nearly a full second after striking Herzberg, Vasquez hit the brakes. And then top it off with systemic issues around the backup driver not actually being ready to react.
  • gambiting 7y ago
    >>> 5.6 seconds before impact, it classified her as a vehicle. Then it changed its mind to “other,” then to vehicle again, back to “other,” then to bicycle, then to “other” again, and finally back to bicycle. This is exactly why I keep saying that autonomous vehicles are not 10 or even 20 years away. More like 50-100 years away if that. Same reason as to why famously a group of researchers in the 60s thought that solving computer recognition of objects would take few months at max, and yet in 2019 our best algorithms still think that a sofa covered in a zebra print is actually a zebra with 99% confidence. Had a human been actually paying attention to the road, I can bet they would start breaking/swearving as soon as they saw something, even if they weren't immediately 100% certain that it's a human - a computer won't until it's 99%+ certain, which is too risky assumption considering the state of visual recognition of objects.
  • 7y ago
• dvdhnt 7y ago
  Personally, I think using automobiles as a blueprint for autonomous transportation is horrifying and preposterous. The first auto I know of went 10mph in 1886. Engineers have had over 100 years to iterate on the concept yet people are still dying. Somehow, Ubers and Googles think they can distill this kind of refinement into a decade or two of software "engineering". Insanity. We should focus on transporting goods and services autonomously at a fraction of the speeds. It should be about efficiency and impact, not getting one human from A to B.
• petercooper 7y ago
  I've long wondered how autonomous vehicles could ever work in the UK, and mostly because there are lots of driving situations I encounter where you have to force your way out or twist the rules to make any sort of progress. Now we can add pedestrians to that which in the UK is a tricky topic since pedestrians have right of way if they're already crossing a side street you're turning into. Even an astute driver can run into trouble here and has to be extremely aware of pedestrian movement.
  • umanwizard 7y ago
    Even in the US, driving is more difficult than it is in Arizona suburbs. The fact that self-driving cars can’t even handle super easy mode makes me very pessimistic that they will ever be used in more than a tiny minority of places.
• JohnClark1337 7y ago
  Just a minor glitch. We'll have that fixed with the next update.
• nix0n 7y ago
  This is murder. Uber wants to increase ridership any way they can, and if you're walking then you're not a customer. Since they got away with it in beta, the release version will continue to murder people.
• draw_down 7y ago
  > 5.6 seconds before impact, it classified her as a vehicle. Then it changed its mind to “other,” then to vehicle again, back to “other,” then to bicycle, then to “other” again, and finally back to bicycle. Why does this system continue driving when it can't figure out what is happening?
• Symmetry 7y ago
  Brad Templeton has also written his analysis of what went wrong in the crash. https://www.forbes.com/sites/bradtempleton/2019/11/06/new-ntsb-reports-on-uber-fatality-reveal-major-errors-by-uber/#4d1bff581781 https://www.forbes.com/sites/bradtempleton/2019/11/06/new-nt...
• paggle 7y ago
  These things seem almost certain to me: * Self-driving cars are not going to be here in less than 5 years * The tech is improving fast enough that it will end up working, and not being vaporware. Within 20 years, self-driving cars will be more than 50% of the new car sales, and eventually it will become the norm.
• neilobremski 7y ago
  Holy shit this is scary. I must have been living under a rock but I didn't realize that FULLY-AUTONOMOUS self-driving cars were even LEGAL yet. > The self-driving car was fully autonomous at the time at the accident, though it had a human driver at the wheel. An internal camera caught the Uber worker looking down and away from the road moments before the accident, unaware of Herzberg’s presence before it was too late. Uber worker ... employee? Were they doing a live test? > ... the team at Uber Advanced Technologies Group has adopted critical program improvements to further prioritize safety ... I certainly hope this means having their terminator vehicles on a closed track instead of out and about. What the hell does "prioritize" have to do with "common sense"? If you throw an unhandled exception ... STOP THE CAR
• Animats 7y ago
  Actual NTSB report: [1] [1] https://dms.ntsb.gov/public/62500-62999/62978/629713.pdf https://dms.ntsb.gov/public/62500-62999/62978/629713.pdf
• csommers 7y ago
  The PMs, engineers, and person sitting at the wheel should all be charged with manslaughter. Maybe then we'd get actual improvements. But who am I kidding? We can't even lock up bankers for simple shit.
• darepublic 7y ago
  Pedestrian detection outside of crosswalk is an edge case, it's a nice to have!
• mark-r 7y ago
  My biggest fear is that accidents of this type will result in the engineers running back to their ML to add another case to the training data. We'll end up playing a game of whack-a-mole (literally?) as new special cases come up. Will training the system to recognize a pedestrian pushing a bicycle enable it to recognize someone riding a mountain bike? Consider why the auto-braking system wasn't enabled. It's because the system can't identify which elements in the environment are harmless, causing too many false positive braking events. The opposite problem is no easier to solve.
• sonthonax 7y ago
  Another thing that seems wrong is that the backup drivers are on their own. Anyone who's done night time guard duty will know how hard it can be to stay awake, let alone maintain attention. It would make so much sense t have a backup driver and co-backup driver (and maybe allow neither a smart phone).
• sitkack 7y ago
  Did they tell the safety drivers this fact?
• more_corn 7y ago
  I can personally attest that they couldn't see you IN the crosswalk either. One nearly ran me down because it failed to yield to pedestrians in the crosswalk when turning right. While I was standing there talking about it, the driver looped the car around the block and tried again. It did the exact same thing the second time. One of the people it happened to walked in to the office to report the bug. The Uber engineers there poo-pooed the concern and never did anything to fix it. This was before they killed the pedestrian. The fact that they utterly failed to create adequate safety systems when repeatedly warned, shows they are absolutely not capable of doing this safely.
  • jacquesm 7y ago
    Oh wow. That's really bad, one of the ground rules of traffic participation is that traffic that goes straight on the same road as you when you turn always has priority. > The Uber engineers there poo-pooed the concern and never did anything to fix it. This is yet another nail in the coffin for Uber and self driving. I think as a company Uber is categorically and institutionally unable to participate in this space in a responsible manner. Their whole corporate culture is totally opposed to what is required to make this a reality.
• basicplus2 7y ago
  it can only be as good as who ever programmed it Consumers and even regulators seem to think somehow software magically adapt to anything. These sort of software environments need standards developed that deal with identified situations that it must satisfy in standardised testing as a starting point.
• desc 7y ago
  tl;dr: Uber have not the slightest idea what they're doing, and should not be permitted to develop anything safety-critical until they can properly explain 'object permanence'. Also, would probably be a good idea to hold the creators, trainers and marketers of anything called 'AI' legally responsible (jointly and severally) for all of such a system's decisions for the next few decades. Might encourage some much-needed caution.
• RickJWagner 7y ago
  Wow. Somebody really dropped the ball on that one. It must be stuff so complicated that it's difficult to see the overview. How else could someone not figure on people outside of crosswalks?
• lightedman 7y ago
  Relevant: https://youtu.be/dnioHfg1xbQ https://youtu.be/dnioHfg1xbQ Tesla cars can't even figure out where they are half of the time.
• KKKKkkkk1 7y ago
  Uber's researchers make very confident presentations about the very advanced ideas that are ostensibly implemented in their cars. How does this square with the apparently very primitive system that's described in the NHTSA report? Example: Jeff Schneider: Self Driving Cars and AI https://youtu.be/jTio_MPQRYc https://youtu.be/jTio_MPQRYc Or: https://eng.uber.com/research/predicting-motion-of-vulnerable-road-users-using-high-definition-maps-and-efficient-convnets/ https://eng.uber.com/research/predicting-motion-of-vulnerabl... Predicting Motion of Vulnerable Road Usersusing High-Definition Maps and Efficient ConvNets Following detection and tracking of traffic actors, prediction of their future motion is the next critical component of a self-driving vehicle (SDV), allowing the SDV to move safely and efficiently in its environment. This is particularly important when it comes to vulnerable road users (VRUs), such as pedestrians and bicyclists. We present a deep learning method for predicting VRU movement where we rasterize high-definition maps and actor’s surroundings into bird’s-eye view image used as input to convolutional networks. In addition, we propose a fast architecture suitable for real-time inference, and present an ablation study of rasterization choices.
• jariel 7y ago
  So is this homicide then?
• dingo_bat 7y ago
  So basically there super advanced system uses a combination of radar and lidar to detect an object 5s before impact and conitunes to track it and predict it's motion on a millisecond-by-millisecond basis. Then when it decides that this is indeed a hazardous situation needing emergency braking it just does nothing for 1000ms? And this was programmed intentionally? Is this a fucking joke? Somebody died due to this stupidity.
• cellular 7y ago
  I can see jaywalking becoming a thing of the past!
• spease 7y ago
  Who knows, maybe the next set of victims will be people jaywalking to get to an Uber, that then get mowed down by another Uber driving at 39mph which classified them as bicycle / other.