28 ms·

Uber disabled Volvo SUV's safety system before fatality

237 points by molecule 9y ago

kelnos 9y ago
It seems a little weird to focus on this. The article itself admits it may be standard practice for makers of self-driving tech to disable any built-in automated driving features in the car. This makes sense: presumably there's no good way to talk to the built-in system; having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe. Even if communication between the two were possible, I'd still imagine that just having a single system operating the care would be much more desirable.
- OrganicMSG 9y ago
  >having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe. Even if communication between the two were possible, I'd still imagine that just having a single system operating the care would be much more desirable. Multiple redundant systems that do not need to talk to each other to coordinate is generally safest. And if the existing systems can not be coordinated with and can make cars less safe because of this, then how are drivers managing it?
- eklavyaa 9y ago
  I wonder why they bought volvo's when they don't wanted to use there radar system or were not able to use the radar system. And is it that easy to disable such an important feature which Volvo brag about ?
  - leetcrew 9y ago
    imagine i'm considering buying 24,000 nice cars from you, but i need to be able to disable certain systems. i feel like you would help me out.
    - eklavyaa 9y ago
      but if I don't want that feature then my need of disabling is superficial
- gambiting 9y ago
  >>having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe. I think this case clearly demonstrates otherwise. Think of the factory fitted as fail-safe - if their own autonomous tech fails(like it did here) you have the one last system that will prevent collision(and it looks almost certain that it would have done that here).
- lmilcin 9y ago
  The problem is that the standard, "dumb", security system is still more effective that "self"-driving solution from Uber. How is a "self"-driving solution allowed to drive in real traffic when it can't break in a simple situation where relatively simple, existing systems can already reliably break.
  - chronolitus 9y ago
    s:break:brake:g
    - lmilcin 9y ago
      Thank you! I am Polish living in Poland and still learning.
- Lazare 9y ago
  > It seems a little weird to focus on this. I'm not sure I'd say that this is being "focused" on. It's an interesting bit of info, the other players in the space (Intel, Volvo, Aptiv) have an interest in getting their spin out there, readers are interested, so we get this article and everyone is happy. It's not a super critical key to the story, but it does help explain what happened in some small way. > having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe. That seems like a bit of a leap to me. The Volvo system is not super-advanced; the relevant bit here would probably be their "Collision Warning with Auto Brake" feature. Multiple independent systems with the ability to sound an alarm and slam on the brakes if they detect an imminent pedestrian collision...seem like they probably would be safer overall? They'd certainly be safer for the pedestrians, and we're currently at 1 pedestrian fatality and 0 fatalities for "people in cars killed due to overzealous/conflicting automated safety systems". I rather suspect Uber disabled it because they wanted better data, not because they thought it was safer for other people on the road. That's certainly a valid choice, but one which might have been wiser if their tech was a lot more reliable.
  - majewsky 9y ago
    > we're currently at 1 pedestrian fatality and 0 fatalities for "people in cars killed due to overzealous/conflicting automated safety systems". Not trying to take sides here, but the last part strikes me as "citation needed".
    - Lazare 9y ago
      Sorry, I guess I left out the implicit "as far as I know". Do you know of anyone who's died that way?
      - dotsh 9y ago
        Yes: https://www.theguardian.com/technology/2016/jun/30/tesla-autopilot-death-self-driving-car-elon-musk https://www.theguardian.com/technology/2016/jun/30/tesla-aut...
- romwell 9y ago
  >having a second system running that can't coordinate with the self-driving system ... is called redundancy and is the basis of building mission-critical, safe systems.
  - TomMarius 9y ago
    No, redundancy means making a backup of your primary system so it can take over, not having a completely different uncooperative system active at the same time.
    - romwell 9y ago
      This is just one of many kinds of redundancy. Please see [1] for several ways in which redundancy is manifested (e.g. triple modular redundancy). Specifically, uncooperative systems running at the same time was exactly how fault tolerance was achieved on the space shuttle[2]: >For true redundancy to take place, all computers must listen to all traffic on all buses, even though they might be commanding just a few. That way they know about all the data generated in the current phase. They must also be processing that data at the same time the other computers do. If there is a failure [...], since the Shuttle's other three computers are sending apparently correct commands to their actuators, the failed computer's commands are physically out-voted That's just one example. There's no need for the definition of the word "redundancy" to be as narrow as you said. [1]https://en.wikipedia.org/wiki/Redundancy_(engineering) https://en.wikipedia.org/wiki/Redundancy_(engineering) [2]https://history.nasa.gov/computers/Ch4-4.html https://history.nasa.gov/computers/Ch4-4.html
      - TomMarius 9y ago
        Redundancy definitely does not mean running two different conflicting systems. Three (identical) - that's a different story.
        simion314 9y ago
        Why not, I have 2 different systems that detect an imminent impact in a wall, then if the systems do not agree you stop, if your main system is experimental then having the backup system that is using different tech then it seems obvious that is safer to stop, then analyze the cause of the conflict.
- couchand 9y ago
  > having a second system running that can't coordinate with the self-driving system It seems like "not running into stuff" is about the only level of coordination necessary here.
  - Piskvorrr 9y ago
    That's a goal which is easy to formulate, but hard to put into code.
    - iainmerrick 9y ago
      The code exists! The collision detection and emergency braking feature which many new cars already have, and which (I assume and hope) has been heavily road-tested.
      - Piskvorrr 9y ago
        That's what has been said of Uber's code as well. Hope-based safety doesn't seem to be working.
        iainmerrick 9y ago
        But my point is, Uber’s self-driving car has lots of goals -- drive safely, drive legally, drive quickly, drive comfortably, go in the right direction. A low-level safety system has a much simpler requirement: if it looks like the car is about to hit something, hit the brakes. And this second category of code is something that already exists, deployed in production today by multiple car manufacturers. That’s the bit Uber turned off. “It’s a hard problem!” doesn’t seem like a good explanation.
    - JorgeGT 9y ago
      The human driver can't coordinate with the self-driving system either, he just makes overriding inputs. Treat the built-in safety system as another overriding actor.
    - couchand 9y ago
- ealexhudson 9y ago
  I don't think it's weird. I called this a few days: https://news.ycombinator.com/item?id=16658566 https://news.ycombinator.com/item?id=16658566 Having them disable a standard feature isn't per se a problem. However, it is an issue if the car is demonstrably less safe with it removed. It's possible that this car was going too fast for this feature to engage anyway (depends on the age of the XC90: if it was post-2014, it should have kicked in). The Volvo system uses a LIDAR just like the main Uber system. In principle they should have been able to replicate this safety feature pretty much precisely. The question then arises; if they're making modifications on systems like that, what other modifications have they made? Are the modifications correct? The issue with the headlamp angle implies there could be other problems. Modern cars are incredibly complex; pulling out single features might be OK when they are add-ons, but the XC90 ships with a lot of this stuff in every model.
  - deleted 9y ago
    [deleted]
- neilalexander 9y ago
  > presumably there's no good way to talk to the built-in system; having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe. Honestly, this is a cop-out. A substantial amount of information is available from the vehicle's CAN networks - including speed and braking action! If the autonomous driving system detects sudden and unplanned changes in speed and/or braking (perhaps caused either by the driver, an external safety system or some kind of impact with something on the road) then the obvious choice there is for the autonomous driving system to release control of the vehicle and let whatever existing systems in the car, or even the driver, handle what is happening. The car's built-in safety systems are a) simpler, b) less likely to malfunction and c) substantially better road-tested. There was no reason whatsoever to disable them.
  - rocqua 9y ago
    If the volvo system were to kick in and brake, the uber system shouldn't relinquish all control. It should only give up the brakes, if you are in the middle of a corner, you'd want the self-driving system to keep steering.
  - davrosthedalek 9y ago
    The only reason I could imagine if the built-in system interferes with the system-under-test. For example, if the built-in LIDAR messes up the LIDAR of the AI.
    - tzakrajs 9y ago
      There is no way Volvo is equipping cars with LIDAR for this kind of use case as radar will do for much much cheaper.
- HenryBemis 9y ago
  It would make sense that some Volvo's systems were disabled, only to be replaced by Uber's central Computer (or whatever is called). This way there won't be 2 breaking systems competing with one-another. Different systems controlling the same end-points is never a good idea.
  - vegardx 9y ago
    Except for emergency systems. If you have two or more independent systems that are able to stop, should any of them should detect an anomaly, you are far more likely to survive a partial system failure. The end result of all systems identifying the problem is the same. In a modern car there are plenty of systems that independently will start. An example is some cars that are fixed to prepare your breaks if you take your foot off the accelerator quickly, more or less giving you instant breaking. At the same time, completely independent of this, emergency breaking might already have been started by the collision detection system.
  - JorgeGT 9y ago
    > there won't be 2 breaking systems competing with one-another. Different systems controlling the same end-points is never a good idea. Aerospace engineer here. Different systems controlling the same end-points is literally a basic engineering safety technique. Not only you have the human pilot as a competing system controlling the same end-points as the autopilot, you may have even different competing autopilots. I had a professor at the Uni who was part of the engineering team of the flight computer of a certain jet fighter. Two engineering teams were given the specs of the system and told to produce a flight computer. The two teams weren't allowed to talk to each other to prevent bugs cross-pollinating the two systems. After all, what good is to have two identical system for redundancy if they can be affected by the same bug? (That's also why pilots eat different meals in-flight!) Then, the two completely different systems are installed, duplicated, into the plane and put to the test. The four computers must perform equally as per the specs. If the recommended output of a single computer diverges from the other three, that computer is faulty and disregarded. If two computers of the same design have a bug and start giving out-of-envelope readings but the other two are consistent, control is handled to them. I'm certainly very appalled to see in this thread how safety engineering techniques that every engineer learns in University is completely disregarded or even reversed. At least in the aerospace community, development and implementation of these safety techniques has been paid in blood. These are lessons that shouldn't be forgotten when developing other transport technologies IMHO.
- iainmerrick 9y ago
  If there's a human driving the car, it's safer to have both the human driver and the automatic emergency stop system. Exactly the same deal if there's an AI driver.
- autogeek 9y ago
  > to disable any built-in automated driving features in the car Automatic Emergency Braking (AEB), which is the system that could have prevented this accident, is not an automated driving function though; it is an active safety function. As such, it is more common to stack such functions. For instance, ABS > AEB > automated driving, with the left most being the function with the highest priority, i.e., it can override the functions to the right. This is how most safety systems in cars work in production.
- Fomite 9y ago
  I can see it from the perspective of Volvo pushing this - one of their cars is now widely shown in the media killing someone, and their whole reputation is safety. I'd also want people to know that Uber was tinkering with the standard safety features, even if it was for good reason.
- Ntrails 9y ago
  > This makes sense: presumably there's no good way to talk to the built-in system; having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe. Even if communication between the two were possible, I'd still imagine that just having a single system operating the care would be much more desirable. That's the exact argument I make for never buying a car with that technology because I'm driving it and I have no way to communicate with it... I'd be super happy to be driven around in an autonomous car, but have zero interest in auto-lane-keeping-pilot-collision avoidance-parking etc
- raverbashing 9y ago
  While it could make it less safe in very specific circumstances, note, it's the same as with a human driver The system kicks in when there is a high probability of a collision ahead. Think of it as the instructor in the driving school that slams the brakes when you're doing something wrong. So basically, the driver (be either human or another system) did something wrong already if that system kicks in
- ilamont 9y ago
  “We don’t want people to be confused or think it was a failure of the technology that we supply for Volvo, because that’s not the case,” Zach Peterson, a spokesman for Aptiv Plc, said by phone. The Volvo XC90’s standard advanced driver-assistance system “has nothing to do” with the Uber test vehicle’s autonomous driving system, he said. Aptiv is speaking up for its technology to avoid being tainted by the fatality involving Uber ... No. This is Volvo pressuring Aptiv to make a statement on its behalf, to avoid being tainted by this PR disaster and to shift investigators' focus away from both companies. Also, in all of the discussions about this incident, let's forget that Elaine Herzberg, a mother of two children, lost her life due to an engineering experiment and publicity stunt ("Come to Arizona! We don't have safety regs!") being conducted on public roads with great risk to members of the public. Volvo, Aptiv, and other participants in these trials should have never signed off ... yet they willingly participated, and an innocent member of the public has paid with her life.
  - la_oveja 9y ago
    Yes, it's tragic. but she crossed the road where she shouldn't. She made a mistake, and a tragic indicent happened. And like this, everyday.
    - lambda 9y ago
      There were multiple failures here. Let's not put all of the blame on the victim. This particular location has the sidewalk end on one side, and there is a paved section in the median which looks like it is intended for crossing. There is a small sign that indicates that you should not cross here. From the video, we can see that while there were street lights on the road, the area in which the apparent crossing existed was completely dark. A pedestrian may not have been able to see the sign indicating that you shouldn't cross here; or even if she did, there are so many roads that are built in pedestrian unfriendly ways that sometimes you just have to cross at unmarked crossings to avoid having to go substantially out of your way. Additionally, the video shows that the driver was distracted immediately before the crash. While the dash cam video seems to indicate that you can't see the pedestrian until it was probably already too late, human eyes are a lot more sensitive over a greater dynamic range than most video cameras, so there's a good chance that if the driver were paying attention, she would have seen the pedestrian before the car did. The thing about safety systems and standards are that there should be some redundancy in them. People make mistakes, so you shouldn't have to rely on every actor acting perfectly to remain safe. In this case, there was a failure on many parts leading to a tragedy. It's definitely worth investigating whether Uber had done enough to ensure that operators will be staying alert during the entire test, and whether Uber had done enough testing before being on open roads of small, possibly moving obstacles on dark parts of the road, like pedestrians, deer, and so on. I'd also put a good amount of blame on whatever department designed that portion of road and sidewalk.
- cptskippy 9y ago
  Most industrial controls have safety interlocks which are independent systems with hard limits. Why is it unrealistic to think that an AI system couldn't be trained to work with-in the limits imposed by these safety systems? If the AI system is being interrupted by the independent safety systems in the car the the AI is doing something wrong.
- pwthornton 9y ago
  It's not weird at all. The point is that Uber should only be disabling these safety features on public roads if their technology is superior. If their technology is less safe, and in this case it appears to be that it is, they shouldn't be driving around on public roads. Keep it to test tracks. Volvo's system would have either stopped or slowed enough to make this crash non-fatal. That's worth pointing out. There is no point in Uber testing technology that is worse than what you can buy any number of cars today.
  - FussyZeus 9y ago
    ...playing Devil's Advocate, as GP said, it's common practice when testing any self driving tech to disable these features on the cars in question so your automated system isn't fighting with another set of automated systems trying to control the same vehicle. That sounds far more unsafe and unpredictable than just disabling the built in ones. Personally I think it's naive and unreasonable to think we're going to get totally safe and rock solid self-driving tech without at least a few coffins worth of people who gave their lives to show us how it could go wrong. There isn't one safety system in existence that wasn't built on top of a pile of corpses.
    - polack 9y ago
      I've had a car with automatic breaking system for a while now and my experience is that they do not kick in easily. Had to test it with a friend that throw an inflatable mattress in front of the car to make it react. It should not be too hard to make these systems co-exist. If one of the systems says break then the car will break. The few times the car breaks when it really didn't need to should be an acceptable side-effect during testing.
      - tzakrajs 9y ago
        Exactly. People are acting like these self-driving programs are so fragile that they can't handle a second system rarely taking control.
      - FussyZeus 9y ago
- mannykannot 9y ago
  Only yesterday there was this discussion [1], in which the general consensus was that the safe way to do things is to put the 'smart' goal-oriented driving stuff on top of the 'dumber-but-reliable' technology to handle specific dangers. It is the latter that has produced most of the statistical evidence justifying the claim that automation makes us safer. I am very skeptical of your claim that this is a less safe way of doing things. There is, in fact, communication between the two systems: the safety system 'says' to the driving system, "no you don't", and that should not be up for negotiation any more than it is when a human is driving. As was pointed out in the other discussion, if the driving system cannot find a way to work within the restrictions of the safety systems, it is doing it wrong. [1] https://news.ycombinator.com/item?id=16681611 https://news.ycombinator.com/item?id=16681611
  - dharmab 9y ago
    You could also record the decisions of the self-driving system and replay the scenario in a simulation environment (either virtual or closed course). Keep the stock safety features on for public roads and still get useful data.
  - thrownblown 9y ago
    Isn't this how the human control systems are? Without reading the referenced convo, I can see many parallels. Like we have reflexes that we don't have full control over, that often don't even interact with our brain but rather are initiated from the spinal cord, which I would classify as dumb in this instance. And layered on top are emotional responses, memories and finally consiousness as, arguably, the control layer.
  - kelnos 9y ago
    Having a dumber system underneath the smarter system is a great idea. I'm not arguing against that. I'm arguing against having two separate systems that are unrelated and are built (likely) without a deep understanding of how the other behaves. Uber themselves should have a dumber system built underneath the smarter system (and maybe they do!); that way they understand all the variables that govern the operation of both and can be reasonably sure that the two systems won't fight each other. They can also build an interlock that causes the smarter system to completely disengage if the dumber system has to take over. I of course don't know this for a fact, but I wouldn't be surprised if Uber doesn't and can't have a complete understanding of how the Volvo system works. Disabling it makes sense. Again, they should replace it with their own dumber system, but that's a separate issue.
- amluto 9y ago
  > Even if communication between the two were possible, I'd still imagine that just having a single system operating the care would be much more desirable. If I had a Volvo XC90, I would want to operate it with two systems: the collision avoidance features plus an advanced system of muscles, eyeballs and brain. The whole point of collision avoidance is to operate in conjunction with a driver. Self-driving should be more or less just another driver.
- tzakrajs 9y ago
  Aren't humans glorified self-driving systems? The second system is a backup for when the human makes a terrible mistake, similar to how this could've been a backup for the computerized self-driving system.
- fixermark 9y ago
  It reads like a story press-released by Volvo to make sure Volvo is CYA'd in this in the public eye (lest uninformed consumers find themselves wondering how good Volvo's collision avoidance is if it couldn't stop that Volvo-branded car from hitting that woman in Arizona).
nwrk 9y ago
There you go. Was seeing the videos of other drivers passing same road. Even the Google XL is better quality than Uber cam. Also, all the lidar gimmick looks like not worked well there. Like the Intel statement from article. Hope there will be some sort of requirements for this equipment defined. Or technically benchmarked before allowed on road. > Intel Corp.’s Mobileye, which makes chips and sensors used in collision-avoidance systems and is a supplier to Aptiv, said Monday that it tested its own software after the crash by playing a video of the Uber incident on a television monitor. Mobileye said it was able to detect Herzberg one second before impact in its internal tests, despite the poor second-hand quality of the video relative to a direct connection to cameras equipped to the car.
- kevin_thibedeau 9y ago
  Nice try. Tesla already demonstrated Mobileye's inability to handle impending collisions.
  - DannyBee 9y ago
    ??? Mobileye explicitly did not want Tesla using them in the fashion they were, and discontinued supplying them with chips. https://www.google.com/search?q=mobileye+sends+tesla+a+letter&rlz=1C5CHFA_enUS770US770&oq=mobileye+sends+tesla+a+letter&aqs=chrome..69i57.5718j0j7&sourceid=chrome&ie=UTF-8 https://www.google.com/search?q=mobileye+sends+tesla+a+lette... This came about because Mobileye was honest about their capabilities: https://electrek.co/2016/07/01/tesla-autopilot-mobileye-fatal-crash-comment/ https://electrek.co/2016/07/01/tesla-autopilot-mobileye-fata... Tesla, in their immediate response (same article), basically tried to imply they have magic abilities to do much better, which, based on later crashes, we know is not true. Mobileye, in response, chose to end their partnership with Tesla (and yes, it was Mobileye who did it. The letter they sent ending the contract can be found if you browse around, after Tesla explicitly asked them to continue and signed a multi-year contract earlier in 2016)
smogcutter 9y ago
This seems like an obvious thing to do: if you’re trying to test the AI, remove complicating, possibly contradictory OEM systems. But compare with the other discussion currently on the front page, where users point out system subsumption is a basic safety principle. You need a safety system to fall back on if the AI fails (as it will, at least at this stage). It seems grossly negligent if Uber didn’t re-implement collision avoidance at a lower level. Edit: link to discussion I’m thinking of https://news.ycombinator.com/item?id=16681611 https://news.ycombinator.com/item?id=16681611
- slobotron 9y ago
  Does not seem obvious to me at all. First thought I had when hearing of the accident was "how come Volvo didn't break, they advertised the tech quite a bit few years back?"
  - deleted 9y ago
    [deleted]
  - smogcutter 9y ago
    Sure, because as a responsible person you’re more interested in safety than training a neural net.
- freddie_mercury 9y ago
  Why is "car's built-in system took control" harder to handle than "human safety driver took control"? Seems like they are exactly the same: "external system took control".
  - buzer 9y ago
    I would imagine it's easier to hook into human operated controls than to car's internal systems. Just put some sensors on steering wheel, pedals, handbrake and couple of others and you will know when someone uses them. If they are used, give up control of the car (or the specific system that was controlled). Getting information that some automated system initiated some operation is almost certainly harder. System only knows that the car is not behaving as expected, but it doesn't know what caused it. You could probably figure out how to determine if something like collision avoidance was triggered (large shifts in expected vs actual outcomes), but in case of something like ABS you could end up with two systems fighting for the control over brakes in emergency situation.
eklavyaa 9y ago
The autonomous vehicle era is in same stage which once passenger planes were in
billysielu 9y ago
Worst website I've seen in a long time.
CookWithMe 9y ago
Volvo is also manufacturing a car with a pedestrian airbag, the V40: https://support.volvocars.com/uk/cars/Pages/owners-manual.aspx?mc=Y555&my=2015&sw=14w46&article=7fceb4e7544b4fbbc0a801e800b0ef6b# https://support.volvocars.com/uk/cars/Pages/owners-manual.as... Both Volvo and Uber could have opted to use this car instead. Even choosing a SUV is question-worthy, as they're known to be more dangerous for pedestrians due to higher bumper heights: https://en.wikipedia.org/wiki/Criticism_of_sport_utility_vehicles#Risk_to_other_road_users https://en.wikipedia.org/wiki/Criticism_of_sport_utility_veh... IMO there is at least some negligence on their part for not choosing a car that is more likely to protect pedestrians.
- autogeek 9y ago
  The pedestrian airbag was introduced before the Automatic Emergency Braking (AEB) safety function was available. Volvo no longer offers the pedestrian airbag as they found most pedestrian accidents were avoided by the AEB. The V40 is also a rather old car and is based on an old platform. The XC90 on the other hand was their newest car at the time the Uber deal was made and is based on their latest platform. So it is not unusual that both Uber and Volvo would prefer the XC90 over the V40. Besides, given the newer technology in the XC90 it is quite possible it is better in pedestrian accidents than the V40 (with the safety systems enabled on both).
andrewprock 9y ago
Why was this being tested at night? It looks like the nighttime visibility was less than 50 feet.
- tzs 9y ago
  The low beam headlights on that model car give just under 250 feet of visibility. At 38 mph that is a little over 4 seconds. That should be fine.
- romwell 9y ago
  I'd say because night time visibility in Arizona on a clear day is way, way more than 50 feet, and the low-quality dashcam footage you see here is highly misleading. Low visibility on a clear day is simply lack of illumination, and it is a horrible excuse, because: 1. The car is equipped with headlights, which solve the "limited night time visibility" problem; 2. The car is equipped with multiple types of high-quality cameras; 3. The car is equipped with a LIDAR, which is an active sensing system. Finally, if one accepts that the visibility was limited to 50 feet, as the video misleadingly suggests, THEN one has to concede that the car was going too fast for how much it could see. People are taught to drive so that at least three seconds worth of road ahead can be seen. If you believe that either the car or the driver were seeing the equivalent of what the webcam footage was showing, the car was going unsafely fast. Hanlon's razor would suggest, though, that the car simply wasn't able to process the data available to it. We'll know for sure if the investigation is successful.
msoad 9y ago
The released video is suspiciously dark. That street has street lighting. Can it be edited?
- romwell 9y ago
  It doesn't even need to be edited to be misleading. Simply taking a crappy dashcam video with low dynamic range suffices; it will make an illusion that anything not directly in the headlights of the vehicle is invisible. I don't know how people can believe a single word of the early official statements which claimed that the victim "suddenly" appeared in front of the vehicle. The problem of "it's too dark to drive" has been solved a hundred years ago with the invention of headlights. Here's an experiment anyone can try: turn on the headlights on a dark road, and see if you really see nothing in the lane to your left (driver side). Whatever you do see (which, under normal circumstances would be at least three seconds worth of road ahead) is the way in which the released video is misleading.
  - kmonsen 9y ago
    Also, if it is too dark to drive at that speed you are supposed to slow down not use it as an excuse after the fact. The speed limit is the max limit, conditions permitting.
- Animats 9y ago
  The video is from a third-party dashcam, not the vehicle sensors, says the NTSB.
derekp7 9y ago
I was wondering why it is necessary for self driving tech to be tested in "live" mode, instead of having the software passively log all event data. Then analyze it to see what the software "would have done" compared to a human driver. Then a lot more testing data could be gathered by outfitting random vehicles (taxis, etc) with the tech, and analyze (and further refine the software) around every event where the software and human driver differed in opinion (i.e., whenever the vehicle abruptly changes speed, did the software detect that it should have hit the brakes at / before the same time).
- maxxxxx 9y ago
  I was wondering about that too.
- WalterSear 9y ago
  Like Tesla is doing?
  - SomewhatLikely 9y ago
    I too assumed this is exactly why Tesla pushed out all the hardware years in advance. To collect training data.
    - deleted 9y ago
      [deleted]
- lopmotr 9y ago
  Any feedback from the response of the vehicle to the commands would be absent. So would whatever happens next. If the computer says "steer left a little" but the video keeps going straight, what's the computer supposed to do next?
  - carlmr 9y ago
    You can't really test the control part of the algorithm, but you could test the detection and object recognition parts.
Someone 9y ago
With friends like this, why would Uber need enemies? We now have public statements from both Velodyne (makes the LiDAR that Uber uses) and Aptiv (makes the hardware used in the Volvos that Uber uses) saying “it wasn’t us”. Volvo is the gentleman here, with its “can’t speculate on the cause of the incident” statement.
- fixermark 9y ago
  The gentleman in public, but we don't know if pressure was put on Aptiv to make a statement so Volvo didn't have to. ;)
yason 9y ago
That's quite opposite to how autonomous systems should be built in the first place: start with isolated modules that do not depend on each other, and stack them on top of each other as the abstraction level goes up. If you have an automatic but a brutal, on-off style emergency-braking module based on, for example, lidar then it's much easier to develop another, maybe less aggressive but much smarter, auto-braking module based on the regular camera inputs. You can count on the failsafe system to act if higher-level modules fail. With braking in particular, maybe even a couple of redundant low-level emergency brake systems would be a good idea before you even consider any of the more high-level systems. You really don't want to hit jaywalking pedestrians or wandering drunks because both kinds do exist in reality. Similarly, once the car has a couple of obstacle-detection and auto-braking systems up and running, it's easier to work on autonomous driving because you can be confident that your computer won't be able to drive the car into anything as these isolated and separate systems will take care to stop the car. Consequently, when you're developing the navigation and routing system it has to be able to rely on the driving functions that deals with the present-time traffic situations, like keeping up with the flow of traffic, slowing down, changing lanes, etc. Again, same thing as with emergency brakes: to develop a smarter higher-level system you need to have lower-level failsafes in place. In the case of autonomous driving a failure should mean a harsh but unnecessary full stop or the inability to continue, not a collision or the inability to stop. They should definitely have left Volvo's own system on as an additional failsafe mechanism.
m3kw9 9y ago
You do need to disable those systems to have a proper test and may have adverse confusion with the AI
ahemphill 9y ago
Warning: this article contains an auto-playing video of the incident.
Zigurd 9y ago
Maybe disabling Volvo's built in autonomous braking capability is needed for operating an autonomous driving system. It seems intuitive that multiple autonomous systems could clash. But it also seems like "It works better than the stock production autonomous braking" would be a gating factor in putting these vehicles on the road.
- kwhitefoot 9y ago
  It's not autonomous it's automatic.

237 points by molecule 9y ago

It seems a little weird to focus on this. The article itself admits it may be standard practice for makers of self-driving tech to disable any built-in automated driving features in the car. This makes sense: presumably there's no good way to talk to the built-in system; having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe. Even if communication between the two were possible, I'd still imagine that just having a single system operating the care would be much more desirable.

OrganicMSG 9y ago
>having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe. Even if communication between the two were possible, I'd still imagine that just having a single system operating the care would be much more desirable. Multiple redundant systems that do not need to talk to each other to coordinate is generally safest. And if the existing systems can not be coordinated with and can make cars less safe because of this, then how are drivers managing it?
eklavyaa 9y ago
I wonder why they bought volvo's when they don't wanted to use there radar system or were not able to use the radar system. And is it that easy to disable such an important feature which Volvo brag about ?
- leetcrew 9y ago
  imagine i'm considering buying 24,000 nice cars from you, but i need to be able to disable certain systems. i feel like you would help me out.
  - eklavyaa 9y ago
    but if I don't want that feature then my need of disabling is superficial
gambiting 9y ago
>>having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe. I think this case clearly demonstrates otherwise. Think of the factory fitted as fail-safe - if their own autonomous tech fails(like it did here) you have the one last system that will prevent collision(and it looks almost certain that it would have done that here).
lmilcin 9y ago
The problem is that the standard, "dumb", security system is still more effective that "self"-driving solution from Uber. How is a "self"-driving solution allowed to drive in real traffic when it can't break in a simple situation where relatively simple, existing systems can already reliably break.
- chronolitus 9y ago
  s:break:brake:g
  - lmilcin 9y ago
    Thank you! I am Polish living in Poland and still learning.
Lazare 9y ago
> It seems a little weird to focus on this. I'm not sure I'd say that this is being "focused" on. It's an interesting bit of info, the other players in the space (Intel, Volvo, Aptiv) have an interest in getting their spin out there, readers are interested, so we get this article and everyone is happy. It's not a super critical key to the story, but it does help explain what happened in some small way. > having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe. That seems like a bit of a leap to me. The Volvo system is not super-advanced; the relevant bit here would probably be their "Collision Warning with Auto Brake" feature. Multiple independent systems with the ability to sound an alarm and slam on the brakes if they detect an imminent pedestrian collision...seem like they probably would be safer overall? They'd certainly be safer for the pedestrians, and we're currently at 1 pedestrian fatality and 0 fatalities for "people in cars killed due to overzealous/conflicting automated safety systems". I rather suspect Uber disabled it because they wanted better data, not because they thought it was safer for other people on the road. That's certainly a valid choice, but one which might have been wiser if their tech was a lot more reliable.
- majewsky 9y ago
  > we're currently at 1 pedestrian fatality and 0 fatalities for "people in cars killed due to overzealous/conflicting automated safety systems". Not trying to take sides here, but the last part strikes me as "citation needed".
  - Lazare 9y ago
    Sorry, I guess I left out the implicit "as far as I know". Do you know of anyone who's died that way?
    - dotsh 9y ago
      Yes: https://www.theguardian.com/technology/2016/jun/30/tesla-autopilot-death-self-driving-car-elon-musk https://www.theguardian.com/technology/2016/jun/30/tesla-aut...
romwell 9y ago
>having a second system running that can't coordinate with the self-driving system ... is called redundancy and is the basis of building mission-critical, safe systems.
- TomMarius 9y ago
  No, redundancy means making a backup of your primary system so it can take over, not having a completely different uncooperative system active at the same time.
  - romwell 9y ago
    This is just one of many kinds of redundancy. Please see [1] for several ways in which redundancy is manifested (e.g. triple modular redundancy). Specifically, uncooperative systems running at the same time was exactly how fault tolerance was achieved on the space shuttle[2]: >For true redundancy to take place, all computers must listen to all traffic on all buses, even though they might be commanding just a few. That way they know about all the data generated in the current phase. They must also be processing that data at the same time the other computers do. If there is a failure [...], since the Shuttle's other three computers are sending apparently correct commands to their actuators, the failed computer's commands are physically out-voted That's just one example. There's no need for the definition of the word "redundancy" to be as narrow as you said. [1]https://en.wikipedia.org/wiki/Redundancy_(engineering) https://en.wikipedia.org/wiki/Redundancy_(engineering) [2]https://history.nasa.gov/computers/Ch4-4.html https://history.nasa.gov/computers/Ch4-4.html
    - TomMarius 9y ago
      Redundancy definitely does not mean running two different conflicting systems. Three (identical) - that's a different story.
      - simion314 9y ago
        Why not, I have 2 different systems that detect an imminent impact in a wall, then if the systems do not agree you stop, if your main system is experimental then having the backup system that is using different tech then it seems obvious that is safer to stop, then analyze the cause of the conflict.
couchand 9y ago
> having a second system running that can't coordinate with the self-driving system It seems like "not running into stuff" is about the only level of coordination necessary here.
- Piskvorrr 9y ago
  That's a goal which is easy to formulate, but hard to put into code.
  - iainmerrick 9y ago
    The code exists! The collision detection and emergency braking feature which many new cars already have, and which (I assume and hope) has been heavily road-tested.
    - Piskvorrr 9y ago
      That's what has been said of Uber's code as well. Hope-based safety doesn't seem to be working.
      - iainmerrick 9y ago
        But my point is, Uber’s self-driving car has lots of goals -- drive safely, drive legally, drive quickly, drive comfortably, go in the right direction. A low-level safety system has a much simpler requirement: if it looks like the car is about to hit something, hit the brakes. And this second category of code is something that already exists, deployed in production today by multiple car manufacturers. That’s the bit Uber turned off. “It’s a hard problem!” doesn’t seem like a good explanation.
  - JorgeGT 9y ago
    The human driver can't coordinate with the self-driving system either, he just makes overriding inputs. Treat the built-in safety system as another overriding actor.
  - couchand 9y ago
ealexhudson 9y ago
I don't think it's weird. I called this a few days: https://news.ycombinator.com/item?id=16658566 https://news.ycombinator.com/item?id=16658566 Having them disable a standard feature isn't per se a problem. However, it is an issue if the car is demonstrably less safe with it removed. It's possible that this car was going too fast for this feature to engage anyway (depends on the age of the XC90: if it was post-2014, it should have kicked in). The Volvo system uses a LIDAR just like the main Uber system. In principle they should have been able to replicate this safety feature pretty much precisely. The question then arises; if they're making modifications on systems like that, what other modifications have they made? Are the modifications correct? The issue with the headlamp angle implies there could be other problems. Modern cars are incredibly complex; pulling out single features might be OK when they are add-ons, but the XC90 ships with a lot of this stuff in every model.
- deleted 9y ago
  [deleted]
neilalexander 9y ago
> presumably there's no good way to talk to the built-in system; having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe. Honestly, this is a cop-out. A substantial amount of information is available from the vehicle's CAN networks - including speed and braking action! If the autonomous driving system detects sudden and unplanned changes in speed and/or braking (perhaps caused either by the driver, an external safety system or some kind of impact with something on the road) then the obvious choice there is for the autonomous driving system to release control of the vehicle and let whatever existing systems in the car, or even the driver, handle what is happening. The car's built-in safety systems are a) simpler, b) less likely to malfunction and c) substantially better road-tested. There was no reason whatsoever to disable them.
- rocqua 9y ago
  If the volvo system were to kick in and brake, the uber system shouldn't relinquish all control. It should only give up the brakes, if you are in the middle of a corner, you'd want the self-driving system to keep steering.
- davrosthedalek 9y ago
  The only reason I could imagine if the built-in system interferes with the system-under-test. For example, if the built-in LIDAR messes up the LIDAR of the AI.
  - tzakrajs 9y ago
    There is no way Volvo is equipping cars with LIDAR for this kind of use case as radar will do for much much cheaper.
HenryBemis 9y ago
It would make sense that some Volvo's systems were disabled, only to be replaced by Uber's central Computer (or whatever is called). This way there won't be 2 breaking systems competing with one-another. Different systems controlling the same end-points is never a good idea.
- vegardx 9y ago
  Except for emergency systems. If you have two or more independent systems that are able to stop, should any of them should detect an anomaly, you are far more likely to survive a partial system failure. The end result of all systems identifying the problem is the same. In a modern car there are plenty of systems that independently will start. An example is some cars that are fixed to prepare your breaks if you take your foot off the accelerator quickly, more or less giving you instant breaking. At the same time, completely independent of this, emergency breaking might already have been started by the collision detection system.
- JorgeGT 9y ago
  > there won't be 2 breaking systems competing with one-another. Different systems controlling the same end-points is never a good idea. Aerospace engineer here. Different systems controlling the same end-points is literally a basic engineering safety technique. Not only you have the human pilot as a competing system controlling the same end-points as the autopilot, you may have even different competing autopilots. I had a professor at the Uni who was part of the engineering team of the flight computer of a certain jet fighter. Two engineering teams were given the specs of the system and told to produce a flight computer. The two teams weren't allowed to talk to each other to prevent bugs cross-pollinating the two systems. After all, what good is to have two identical system for redundancy if they can be affected by the same bug? (That's also why pilots eat different meals in-flight!) Then, the two completely different systems are installed, duplicated, into the plane and put to the test. The four computers must perform equally as per the specs. If the recommended output of a single computer diverges from the other three, that computer is faulty and disregarded. If two computers of the same design have a bug and start giving out-of-envelope readings but the other two are consistent, control is handled to them. I'm certainly very appalled to see in this thread how safety engineering techniques that every engineer learns in University is completely disregarded or even reversed. At least in the aerospace community, development and implementation of these safety techniques has been paid in blood. These are lessons that shouldn't be forgotten when developing other transport technologies IMHO.
iainmerrick 9y ago
If there's a human driving the car, it's safer to have both the human driver and the automatic emergency stop system. Exactly the same deal if there's an AI driver.
autogeek 9y ago
> to disable any built-in automated driving features in the car Automatic Emergency Braking (AEB), which is the system that could have prevented this accident, is not an automated driving function though; it is an active safety function. As such, it is more common to stack such functions. For instance, ABS > AEB > automated driving, with the left most being the function with the highest priority, i.e., it can override the functions to the right. This is how most safety systems in cars work in production.
Fomite 9y ago
I can see it from the perspective of Volvo pushing this - one of their cars is now widely shown in the media killing someone, and their whole reputation is safety. I'd also want people to know that Uber was tinkering with the standard safety features, even if it was for good reason.
Ntrails 9y ago
> This makes sense: presumably there's no good way to talk to the built-in system; having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe. Even if communication between the two were possible, I'd still imagine that just having a single system operating the care would be much more desirable. That's the exact argument I make for never buying a car with that technology because I'm driving it and I have no way to communicate with it... I'd be super happy to be driven around in an autonomous car, but have zero interest in auto-lane-keeping-pilot-collision avoidance-parking etc
raverbashing 9y ago
While it could make it less safe in very specific circumstances, note, it's the same as with a human driver The system kicks in when there is a high probability of a collision ahead. Think of it as the instructor in the driving school that slams the brakes when you're doing something wrong. So basically, the driver (be either human or another system) did something wrong already if that system kicks in
ilamont 9y ago
“We don’t want people to be confused or think it was a failure of the technology that we supply for Volvo, because that’s not the case,” Zach Peterson, a spokesman for Aptiv Plc, said by phone. The Volvo XC90’s standard advanced driver-assistance system “has nothing to do” with the Uber test vehicle’s autonomous driving system, he said. Aptiv is speaking up for its technology to avoid being tainted by the fatality involving Uber ... No. This is Volvo pressuring Aptiv to make a statement on its behalf, to avoid being tainted by this PR disaster and to shift investigators' focus away from both companies. Also, in all of the discussions about this incident, let's forget that Elaine Herzberg, a mother of two children, lost her life due to an engineering experiment and publicity stunt ("Come to Arizona! We don't have safety regs!") being conducted on public roads with great risk to members of the public. Volvo, Aptiv, and other participants in these trials should have never signed off ... yet they willingly participated, and an innocent member of the public has paid with her life.
- la_oveja 9y ago
  Yes, it's tragic. but she crossed the road where she shouldn't. She made a mistake, and a tragic indicent happened. And like this, everyday.
  - lambda 9y ago
    There were multiple failures here. Let's not put all of the blame on the victim. This particular location has the sidewalk end on one side, and there is a paved section in the median which looks like it is intended for crossing. There is a small sign that indicates that you should not cross here. From the video, we can see that while there were street lights on the road, the area in which the apparent crossing existed was completely dark. A pedestrian may not have been able to see the sign indicating that you shouldn't cross here; or even if she did, there are so many roads that are built in pedestrian unfriendly ways that sometimes you just have to cross at unmarked crossings to avoid having to go substantially out of your way. Additionally, the video shows that the driver was distracted immediately before the crash. While the dash cam video seems to indicate that you can't see the pedestrian until it was probably already too late, human eyes are a lot more sensitive over a greater dynamic range than most video cameras, so there's a good chance that if the driver were paying attention, she would have seen the pedestrian before the car did. The thing about safety systems and standards are that there should be some redundancy in them. People make mistakes, so you shouldn't have to rely on every actor acting perfectly to remain safe. In this case, there was a failure on many parts leading to a tragedy. It's definitely worth investigating whether Uber had done enough to ensure that operators will be staying alert during the entire test, and whether Uber had done enough testing before being on open roads of small, possibly moving obstacles on dark parts of the road, like pedestrians, deer, and so on. I'd also put a good amount of blame on whatever department designed that portion of road and sidewalk.
cptskippy 9y ago
Most industrial controls have safety interlocks which are independent systems with hard limits. Why is it unrealistic to think that an AI system couldn't be trained to work with-in the limits imposed by these safety systems? If the AI system is being interrupted by the independent safety systems in the car the the AI is doing something wrong.
pwthornton 9y ago
It's not weird at all. The point is that Uber should only be disabling these safety features on public roads if their technology is superior. If their technology is less safe, and in this case it appears to be that it is, they shouldn't be driving around on public roads. Keep it to test tracks. Volvo's system would have either stopped or slowed enough to make this crash non-fatal. That's worth pointing out. There is no point in Uber testing technology that is worse than what you can buy any number of cars today.
- FussyZeus 9y ago
  ...playing Devil's Advocate, as GP said, it's common practice when testing any self driving tech to disable these features on the cars in question so your automated system isn't fighting with another set of automated systems trying to control the same vehicle. That sounds far more unsafe and unpredictable than just disabling the built in ones. Personally I think it's naive and unreasonable to think we're going to get totally safe and rock solid self-driving tech without at least a few coffins worth of people who gave their lives to show us how it could go wrong. There isn't one safety system in existence that wasn't built on top of a pile of corpses.
  - polack 9y ago
    I've had a car with automatic breaking system for a while now and my experience is that they do not kick in easily. Had to test it with a friend that throw an inflatable mattress in front of the car to make it react. It should not be too hard to make these systems co-exist. If one of the systems says break then the car will break. The few times the car breaks when it really didn't need to should be an acceptable side-effect during testing.
    - tzakrajs 9y ago
      Exactly. People are acting like these self-driving programs are so fragile that they can't handle a second system rarely taking control.
    - FussyZeus 9y ago
mannykannot 9y ago
Only yesterday there was this discussion [1], in which the general consensus was that the safe way to do things is to put the 'smart' goal-oriented driving stuff on top of the 'dumber-but-reliable' technology to handle specific dangers. It is the latter that has produced most of the statistical evidence justifying the claim that automation makes us safer. I am very skeptical of your claim that this is a less safe way of doing things. There is, in fact, communication between the two systems: the safety system 'says' to the driving system, "no you don't", and that should not be up for negotiation any more than it is when a human is driving. As was pointed out in the other discussion, if the driving system cannot find a way to work within the restrictions of the safety systems, it is doing it wrong. [1] https://news.ycombinator.com/item?id=16681611 https://news.ycombinator.com/item?id=16681611
- dharmab 9y ago
  You could also record the decisions of the self-driving system and replay the scenario in a simulation environment (either virtual or closed course). Keep the stock safety features on for public roads and still get useful data.
- thrownblown 9y ago
  Isn't this how the human control systems are? Without reading the referenced convo, I can see many parallels. Like we have reflexes that we don't have full control over, that often don't even interact with our brain but rather are initiated from the spinal cord, which I would classify as dumb in this instance. And layered on top are emotional responses, memories and finally consiousness as, arguably, the control layer.
- kelnos 9y ago
  Having a dumber system underneath the smarter system is a great idea. I'm not arguing against that. I'm arguing against having two separate systems that are unrelated and are built (likely) without a deep understanding of how the other behaves. Uber themselves should have a dumber system built underneath the smarter system (and maybe they do!); that way they understand all the variables that govern the operation of both and can be reasonably sure that the two systems won't fight each other. They can also build an interlock that causes the smarter system to completely disengage if the dumber system has to take over. I of course don't know this for a fact, but I wouldn't be surprised if Uber doesn't and can't have a complete understanding of how the Volvo system works. Disabling it makes sense. Again, they should replace it with their own dumber system, but that's a separate issue.
amluto 9y ago
> Even if communication between the two were possible, I'd still imagine that just having a single system operating the care would be much more desirable. If I had a Volvo XC90, I would want to operate it with two systems: the collision avoidance features plus an advanced system of muscles, eyeballs and brain. The whole point of collision avoidance is to operate in conjunction with a driver. Self-driving should be more or less just another driver.
tzakrajs 9y ago
Aren't humans glorified self-driving systems? The second system is a backup for when the human makes a terrible mistake, similar to how this could've been a backup for the computerized self-driving system.
fixermark 9y ago
It reads like a story press-released by Volvo to make sure Volvo is CYA'd in this in the public eye (lest uninformed consumers find themselves wondering how good Volvo's collision avoidance is if it couldn't stop that Volvo-branded car from hitting that woman in Arizona).

nwrk 9y ago

There you go. Was seeing the videos of other drivers passing same road. Even the Google XL is better quality than Uber cam. Also, all the lidar gimmick looks like not worked well there. Like the Intel statement from article. Hope there will be some sort of requirements for this equipment defined. Or technically benchmarked before allowed on road. > Intel Corp.’s Mobileye, which makes chips and sensors used in collision-avoidance systems and is a supplier to Aptiv, said Monday that it tested its own software after the crash by playing a video of the Uber incident on a television monitor. Mobileye said it was able to detect Herzberg one second before impact in its internal tests, despite the poor second-hand quality of the video relative to a direct connection to cameras equipped to the car.

kevin_thibedeau 9y ago
Nice try. Tesla already demonstrated Mobileye's inability to handle impending collisions.
- DannyBee 9y ago
  ??? Mobileye explicitly did not want Tesla using them in the fashion they were, and discontinued supplying them with chips. https://www.google.com/search?q=mobileye+sends+tesla+a+letter&rlz=1C5CHFA_enUS770US770&oq=mobileye+sends+tesla+a+letter&aqs=chrome..69i57.5718j0j7&sourceid=chrome&ie=UTF-8 https://www.google.com/search?q=mobileye+sends+tesla+a+lette... This came about because Mobileye was honest about their capabilities: https://electrek.co/2016/07/01/tesla-autopilot-mobileye-fatal-crash-comment/ https://electrek.co/2016/07/01/tesla-autopilot-mobileye-fata... Tesla, in their immediate response (same article), basically tried to imply they have magic abilities to do much better, which, based on later crashes, we know is not true. Mobileye, in response, chose to end their partnership with Tesla (and yes, it was Mobileye who did it. The letter they sent ending the contract can be found if you browse around, after Tesla explicitly asked them to continue and signed a multi-year contract earlier in 2016)

smogcutter 9y ago

This seems like an obvious thing to do: if you’re trying to test the AI, remove complicating, possibly contradictory OEM systems. But compare with the other discussion currently on the front page, where users point out system subsumption is a basic safety principle. You need a safety system to fall back on if the AI fails (as it will, at least at this stage). It seems grossly negligent if Uber didn’t re-implement collision avoidance at a lower level. Edit: link to discussion I’m thinking of https://news.ycombinator.com/item?id=16681611 https://news.ycombinator.com/item?id=16681611

slobotron 9y ago
Does not seem obvious to me at all. First thought I had when hearing of the accident was "how come Volvo didn't break, they advertised the tech quite a bit few years back?"
- deleted 9y ago
  [deleted]
- smogcutter 9y ago
  Sure, because as a responsible person you’re more interested in safety than training a neural net.
freddie_mercury 9y ago
Why is "car's built-in system took control" harder to handle than "human safety driver took control"? Seems like they are exactly the same: "external system took control".
- buzer 9y ago
  I would imagine it's easier to hook into human operated controls than to car's internal systems. Just put some sensors on steering wheel, pedals, handbrake and couple of others and you will know when someone uses them. If they are used, give up control of the car (or the specific system that was controlled). Getting information that some automated system initiated some operation is almost certainly harder. System only knows that the car is not behaving as expected, but it doesn't know what caused it. You could probably figure out how to determine if something like collision avoidance was triggered (large shifts in expected vs actual outcomes), but in case of something like ABS you could end up with two systems fighting for the control over brakes in emergency situation.

eklavyaa 9y ago

The autonomous vehicle era is in same stage which once passenger planes were in

billysielu 9y ago

Worst website I've seen in a long time.

CookWithMe 9y ago

Volvo is also manufacturing a car with a pedestrian airbag, the V40: https://support.volvocars.com/uk/cars/Pages/owners-manual.aspx?mc=Y555&my=2015&sw=14w46&article=7fceb4e7544b4fbbc0a801e800b0ef6b# https://support.volvocars.com/uk/cars/Pages/owners-manual.as... Both Volvo and Uber could have opted to use this car instead. Even choosing a SUV is question-worthy, as they're known to be more dangerous for pedestrians due to higher bumper heights: https://en.wikipedia.org/wiki/Criticism_of_sport_utility_vehicles#Risk_to_other_road_users https://en.wikipedia.org/wiki/Criticism_of_sport_utility_veh... IMO there is at least some negligence on their part for not choosing a car that is more likely to protect pedestrians.

autogeek 9y ago
The pedestrian airbag was introduced before the Automatic Emergency Braking (AEB) safety function was available. Volvo no longer offers the pedestrian airbag as they found most pedestrian accidents were avoided by the AEB. The V40 is also a rather old car and is based on an old platform. The XC90 on the other hand was their newest car at the time the Uber deal was made and is based on their latest platform. So it is not unusual that both Uber and Volvo would prefer the XC90 over the V40. Besides, given the newer technology in the XC90 it is quite possible it is better in pedestrian accidents than the V40 (with the safety systems enabled on both).

andrewprock 9y ago

Why was this being tested at night? It looks like the nighttime visibility was less than 50 feet.

tzs 9y ago
The low beam headlights on that model car give just under 250 feet of visibility. At 38 mph that is a little over 4 seconds. That should be fine.
romwell 9y ago
I'd say because night time visibility in Arizona on a clear day is way, way more than 50 feet, and the low-quality dashcam footage you see here is highly misleading. Low visibility on a clear day is simply lack of illumination, and it is a horrible excuse, because: 1. The car is equipped with headlights, which solve the "limited night time visibility" problem; 2. The car is equipped with multiple types of high-quality cameras; 3. The car is equipped with a LIDAR, which is an active sensing system. Finally, if one accepts that the visibility was limited to 50 feet, as the video misleadingly suggests, THEN one has to concede that the car was going too fast for how much it could see. People are taught to drive so that at least three seconds worth of road ahead can be seen. If you believe that either the car or the driver were seeing the equivalent of what the webcam footage was showing, the car was going unsafely fast. Hanlon's razor would suggest, though, that the car simply wasn't able to process the data available to it. We'll know for sure if the investigation is successful.

msoad 9y ago

The released video is suspiciously dark. That street has street lighting. Can it be edited?

romwell 9y ago
It doesn't even need to be edited to be misleading. Simply taking a crappy dashcam video with low dynamic range suffices; it will make an illusion that anything not directly in the headlights of the vehicle is invisible. I don't know how people can believe a single word of the early official statements which claimed that the victim "suddenly" appeared in front of the vehicle. The problem of "it's too dark to drive" has been solved a hundred years ago with the invention of headlights. Here's an experiment anyone can try: turn on the headlights on a dark road, and see if you really see nothing in the lane to your left (driver side). Whatever you do see (which, under normal circumstances would be at least three seconds worth of road ahead) is the way in which the released video is misleading.
- kmonsen 9y ago
  Also, if it is too dark to drive at that speed you are supposed to slow down not use it as an excuse after the fact. The speed limit is the max limit, conditions permitting.
Animats 9y ago
The video is from a third-party dashcam, not the vehicle sensors, says the NTSB.

derekp7 9y ago

I was wondering why it is necessary for self driving tech to be tested in "live" mode, instead of having the software passively log all event data. Then analyze it to see what the software "would have done" compared to a human driver. Then a lot more testing data could be gathered by outfitting random vehicles (taxis, etc) with the tech, and analyze (and further refine the software) around every event where the software and human driver differed in opinion (i.e., whenever the vehicle abruptly changes speed, did the software detect that it should have hit the brakes at / before the same time).

maxxxxx 9y ago
I was wondering about that too.
WalterSear 9y ago
Like Tesla is doing?
- SomewhatLikely 9y ago
  I too assumed this is exactly why Tesla pushed out all the hardware years in advance. To collect training data.
  - deleted 9y ago
    [deleted]
lopmotr 9y ago
Any feedback from the response of the vehicle to the commands would be absent. So would whatever happens next. If the computer says "steer left a little" but the video keeps going straight, what's the computer supposed to do next?
- carlmr 9y ago
  You can't really test the control part of the algorithm, but you could test the detection and object recognition parts.

Someone 9y ago

With friends like this, why would Uber need enemies? We now have public statements from both Velodyne (makes the LiDAR that Uber uses) and Aptiv (makes the hardware used in the Volvos that Uber uses) saying “it wasn’t us”. Volvo is the gentleman here, with its “can’t speculate on the cause of the incident” statement.

fixermark 9y ago
The gentleman in public, but we don't know if pressure was put on Aptiv to make a statement so Volvo didn't have to. ;)

yason 9y ago

That's quite opposite to how autonomous systems should be built in the first place: start with isolated modules that do not depend on each other, and stack them on top of each other as the abstraction level goes up. If you have an automatic but a brutal, on-off style emergency-braking module based on, for example, lidar then it's much easier to develop another, maybe less aggressive but much smarter, auto-braking module based on the regular camera inputs. You can count on the failsafe system to act if higher-level modules fail. With braking in particular, maybe even a couple of redundant low-level emergency brake systems would be a good idea before you even consider any of the more high-level systems. You really don't want to hit jaywalking pedestrians or wandering drunks because both kinds do exist in reality. Similarly, once the car has a couple of obstacle-detection and auto-braking systems up and running, it's easier to work on autonomous driving because you can be confident that your computer won't be able to drive the car into anything as these isolated and separate systems will take care to stop the car. Consequently, when you're developing the navigation and routing system it has to be able to rely on the driving functions that deals with the present-time traffic situations, like keeping up with the flow of traffic, slowing down, changing lanes, etc. Again, same thing as with emergency brakes: to develop a smarter higher-level system you need to have lower-level failsafes in place. In the case of autonomous driving a failure should mean a harsh but unnecessary full stop or the inability to continue, not a collision or the inability to stop. They should definitely have left Volvo's own system on as an additional failsafe mechanism.

m3kw9 9y ago

You do need to disable those systems to have a proper test and may have adverse confusion with the AI

ahemphill 9y ago

Warning: this article contains an auto-playing video of the incident.

Zigurd 9y ago

Maybe disabling Volvo's built in autonomous braking capability is needed for operating an autonomous driving system. It seems intuitive that multiple autonomous systems could clash. But it also seems like "It works better than the stock production autonomous braking" would be a gating factor in putting these vehicles on the road.

kwhitefoot 9y ago
It's not autonomous it's automatic.