11 ms·
It seems a little weird to focus on this. The article itself admits it may be standard practice for makers of self-driving tech to disable any built-in automat
by kelnos 9y ago
It seems a little weird to focus on this. The article itself admits it may be standard practice for makers of self-driving tech to disable any built-in automated driving features in the car. This makes sense: presumably there's no good way to talk to the built-in system; having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe. Even if communication between the two were possible, I'd still imagine that just having a single system operating the care would be much more desirable.
- OrganicMSG 9y ago>having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe. Even if communication between the two were possible, I'd still imagine that just having a single system operating the care would be much more desirable. Multiple redundant systems that do not need to talk to each other to coordinate is generally safest. And if the existing systems can not be coordinated with and can make cars less safe because of this, then how are drivers managing it?
- eklavyaa 9y agoI wonder why they bought volvo's when they don't wanted to use there radar system or were not able to use the radar system. And is it that easy to disable such an important feature which Volvo brag about ?
- gambiting 9y ago>>having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe. I think this case clearly demonstrates otherwise. Think of the factory fitted as fail-safe - if their own autonomous tech fails(like it did here) you have the one last system that will prevent collision(and it looks almost certain that it would have done that here).
- lmilcin 9y agoThe problem is that the standard, "dumb", security system is still more effective that "self"-driving solution from Uber. How is a "self"-driving solution allowed to drive in real traffic when it can't break in a simple situation where relatively simple, existing systems can already reliably break.
- chronolitus 9y agos:break:brake:g
- lmilcin 9y agoThank you! I am Polish living in Poland and still learning.
- Lazare 9y ago> It seems a little weird to focus on this. I'm not sure I'd say that this is being "focused" on. It's an interesting bit of info, the other players in the space (Intel, Volvo, Aptiv) have an interest in getting their spin out there, readers are interested, so we get this article and everyone is happy. It's not a super critical key to the story, but it does help explain what happened in some small way. > having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe. That seems like a bit of a leap to me. The Volvo system is not super-advanced; the relevant bit here would probably be their "Collision Warning with Auto Brake" feature. Multiple independent systems with the ability to sound an alarm and slam on the brakes if they detect an imminent pedestrian collision...seem like they probably would be safer overall? They'd certainly be safer for the pedestrians, and we're currently at 1 pedestrian fatality and 0 fatalities for "people in cars killed due to overzealous/conflicting automated safety systems". I rather suspect Uber disabled it because they wanted better data, not because they thought it was safer for other people on the road. That's certainly a valid choice, but one which might have been wiser if their tech was a lot more reliable.
- majewsky 9y ago> we're currently at 1 pedestrian fatality and 0 fatalities for "people in cars killed due to overzealous/conflicting automated safety systems". Not trying to take sides here, but the last part strikes me as "citation needed".
- romwell 9y ago>having a second system running that can't coordinate with the self-driving system ... is called redundancy and is the basis of building mission-critical, safe systems.
- TomMarius 9y agoNo, redundancy means making a backup of your primary system so it can take over, not having a completely different uncooperative system active at the same time.
- romwell 9y agoThis is just one of many kinds of redundancy. Please see [1] for several ways in which redundancy is manifested (e.g. triple modular redundancy). Specifically, uncooperative systems running at the same time was exactly how fault tolerance was achieved on the space shuttle[2]: >For true redundancy to take place, all computers must listen to all traffic on all buses, even though they might be commanding just a few. That way they know about all the data generated in the current phase. They must also be processing that data at the same time the other computers do. If there is a failure [...], since the Shuttle's other three computers are sending apparently correct commands to their actuators, the failed computer's commands are physically out-voted That's just one example. There's no need for the definition of the word "redundancy" to be as narrow as you said. [1]https://en.wikipedia.org/wiki/Redundancy_(engineering) https://en.wikipedia.org/wiki/Redundancy_(engineering) [2]https://history.nasa.gov/computers/Ch4-4.html https://history.nasa.gov/computers/Ch4-4.html
- TomMarius 9y agoRedundancy definitely does not mean running two different conflicting systems. Three (identical) - that's a different story.
- simion314 9y agoWhy not, I have 2 different systems that detect an imminent impact in a wall, then if the systems do not agree you stop, if your main system is experimental then having the backup system that is using different tech then it seems obvious that is safer to stop, then analyze the cause of the conflict.
- couchand 9y ago> having a second system running that can't coordinate with the self-driving system It seems like "not running into stuff" is about the only level of coordination necessary here.
- Piskvorrr 9y agoThat's a goal which is easy to formulate, but hard to put into code.
- iainmerrick 9y agoThe code exists! The collision detection and emergency braking feature which many new cars already have, and which (I assume and hope) has been heavily road-tested.
- Piskvorrr 9y agoThat's what has been said of Uber's code as well. Hope-based safety doesn't seem to be working.
- iainmerrick 9y agoBut my point is, Uber’s self-driving car has lots of goals -- drive safely, drive legally, drive quickly, drive comfortably, go in the right direction. A low-level safety system has a much simpler requirement: if it looks like the car is about to hit something, hit the brakes. And this second category of code is something that already exists, deployed in production today by multiple car manufacturers. That’s the bit Uber turned off. “It’s a hard problem!” doesn’t seem like a good explanation.
- JorgeGT 9y agoThe human driver can't coordinate with the self-driving system either, he just makes overriding inputs. Treat the built-in safety system as another overriding actor.
- couchand 9y ago
- ealexhudson 9y agoI don't think it's weird. I called this a few days: https://news.ycombinator.com/item?id=16658566 https://news.ycombinator.com/item?id=16658566 Having them disable a standard feature isn't per se a problem. However, it is an issue if the car is demonstrably less safe with it removed. It's possible that this car was going too fast for this feature to engage anyway (depends on the age of the XC90: if it was post-2014, it should have kicked in). The Volvo system uses a LIDAR just like the main Uber system. In principle they should have been able to replicate this safety feature pretty much precisely. The question then arises; if they're making modifications on systems like that, what other modifications have they made? Are the modifications correct? The issue with the headlamp angle implies there could be other problems. Modern cars are incredibly complex; pulling out single features might be OK when they are add-ons, but the XC90 ships with a lot of this stuff in every model.
- deleted 9y ago[deleted]
- neilalexander 9y ago> presumably there's no good way to talk to the built-in system; having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe. Honestly, this is a cop-out. A substantial amount of information is available from the vehicle's CAN networks - including speed and braking action! If the autonomous driving system detects sudden and unplanned changes in speed and/or braking (perhaps caused either by the driver, an external safety system or some kind of impact with something on the road) then the obvious choice there is for the autonomous driving system to release control of the vehicle and let whatever existing systems in the car, or even the driver, handle what is happening. The car's built-in safety systems are a) simpler, b) less likely to malfunction and c) substantially better road-tested. There was no reason whatsoever to disable them.
- rocqua 9y agoIf the volvo system were to kick in and brake, the uber system shouldn't relinquish all control. It should only give up the brakes, if you are in the middle of a corner, you'd want the self-driving system to keep steering.
- davrosthedalek 9y agoThe only reason I could imagine if the built-in system interferes with the system-under-test. For example, if the built-in LIDAR messes up the LIDAR of the AI.
- tzakrajs 9y agoThere is no way Volvo is equipping cars with LIDAR for this kind of use case as radar will do for much much cheaper.
- HenryBemis 9y agoIt would make sense that some Volvo's systems were disabled, only to be replaced by Uber's central Computer (or whatever is called). This way there won't be 2 breaking systems competing with one-another. Different systems controlling the same end-points is never a good idea.
- vegardx 9y agoExcept for emergency systems. If you have two or more independent systems that are able to stop, should any of them should detect an anomaly, you are far more likely to survive a partial system failure. The end result of all systems identifying the problem is the same. In a modern car there are plenty of systems that independently will start. An example is some cars that are fixed to prepare your breaks if you take your foot off the accelerator quickly, more or less giving you instant breaking. At the same time, completely independent of this, emergency breaking might already have been started by the collision detection system.
- JorgeGT 9y ago> there won't be 2 breaking systems competing with one-another. Different systems controlling the same end-points is never a good idea. Aerospace engineer here. Different systems controlling the same end-points is literally a basic engineering safety technique. Not only you have the human pilot as a competing system controlling the same end-points as the autopilot, you may have even different competing autopilots. I had a professor at the Uni who was part of the engineering team of the flight computer of a certain jet fighter. Two engineering teams were given the specs of the system and told to produce a flight computer. The two teams weren't allowed to talk to each other to prevent bugs cross-pollinating the two systems. After all, what good is to have two identical system for redundancy if they can be affected by the same bug? (That's also why pilots eat different meals in-flight!) Then, the two completely different systems are installed, duplicated, into the plane and put to the test. The four computers must perform equally as per the specs. If the recommended output of a single computer diverges from the other three, that computer is faulty and disregarded. If two computers of the same design have a bug and start giving out-of-envelope readings but the other two are consistent, control is handled to them. I'm certainly very appalled to see in this thread how safety engineering techniques that every engineer learns in University is completely disregarded or even reversed. At least in the aerospace community, development and implementation of these safety techniques has been paid in blood. These are lessons that shouldn't be forgotten when developing other transport technologies IMHO.
- iainmerrick 9y agoIf there's a human driving the car, it's safer to have both the human driver and the automatic emergency stop system. Exactly the same deal if there's an AI driver.
- autogeek 9y ago> to disable any built-in automated driving features in the car Automatic Emergency Braking (AEB), which is the system that could have prevented this accident, is not an automated driving function though; it is an active safety function. As such, it is more common to stack such functions. For instance, ABS > AEB > automated driving, with the left most being the function with the highest priority, i.e., it can override the functions to the right. This is how most safety systems in cars work in production.
- Fomite 9y agoI can see it from the perspective of Volvo pushing this - one of their cars is now widely shown in the media killing someone, and their whole reputation is safety. I'd also want people to know that Uber was tinkering with the standard safety features, even if it was for good reason.
- Ntrails 9y ago> This makes sense: presumably there's no good way to talk to the built-in system; having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe. Even if communication between the two were possible, I'd still imagine that just having a single system operating the care would be much more desirable. That's the exact argument I make for never buying a car with that technology because I'm driving it and I have no way to communicate with it... I'd be super happy to be driven around in an autonomous car, but have zero interest in auto-lane-keeping-pilot-collision avoidance-parking etc
- raverbashing 9y agoWhile it could make it less safe in very specific circumstances, note, it's the same as with a human driver The system kicks in when there is a high probability of a collision ahead. Think of it as the instructor in the driving school that slams the brakes when you're doing something wrong. So basically, the driver (be either human or another system) did something wrong already if that system kicks in
- ilamont 9y ago“We don’t want people to be confused or think it was a failure of the technology that we supply for Volvo, because that’s not the case,” Zach Peterson, a spokesman for Aptiv Plc, said by phone. The Volvo XC90’s standard advanced driver-assistance system “has nothing to do” with the Uber test vehicle’s autonomous driving system, he said. Aptiv is speaking up for its technology to avoid being tainted by the fatality involving Uber ... No. This is Volvo pressuring Aptiv to make a statement on its behalf, to avoid being tainted by this PR disaster and to shift investigators' focus away from both companies. Also, in all of the discussions about this incident, let's forget that Elaine Herzberg, a mother of two children, lost her life due to an engineering experiment and publicity stunt ("Come to Arizona! We don't have safety regs!") being conducted on public roads with great risk to members of the public. Volvo, Aptiv, and other participants in these trials should have never signed off ... yet they willingly participated, and an innocent member of the public has paid with her life.
- la_oveja 9y agoYes, it's tragic. but she crossed the road where she shouldn't. She made a mistake, and a tragic indicent happened. And like this, everyday.
- lambda 9y agoThere were multiple failures here. Let's not put all of the blame on the victim. This particular location has the sidewalk end on one side, and there is a paved section in the median which looks like it is intended for crossing. There is a small sign that indicates that you should not cross here. From the video, we can see that while there were street lights on the road, the area in which the apparent crossing existed was completely dark. A pedestrian may not have been able to see the sign indicating that you shouldn't cross here; or even if she did, there are so many roads that are built in pedestrian unfriendly ways that sometimes you just have to cross at unmarked crossings to avoid having to go substantially out of your way. Additionally, the video shows that the driver was distracted immediately before the crash. While the dash cam video seems to indicate that you can't see the pedestrian until it was probably already too late, human eyes are a lot more sensitive over a greater dynamic range than most video cameras, so there's a good chance that if the driver were paying attention, she would have seen the pedestrian before the car did. The thing about safety systems and standards are that there should be some redundancy in them. People make mistakes, so you shouldn't have to rely on every actor acting perfectly to remain safe. In this case, there was a failure on many parts leading to a tragedy. It's definitely worth investigating whether Uber had done enough to ensure that operators will be staying alert during the entire test, and whether Uber had done enough testing before being on open roads of small, possibly moving obstacles on dark parts of the road, like pedestrians, deer, and so on. I'd also put a good amount of blame on whatever department designed that portion of road and sidewalk.
- cptskippy 9y agoMost industrial controls have safety interlocks which are independent systems with hard limits. Why is it unrealistic to think that an AI system couldn't be trained to work with-in the limits imposed by these safety systems? If the AI system is being interrupted by the independent safety systems in the car the the AI is doing something wrong.
- pwthornton 9y agoIt's not weird at all. The point is that Uber should only be disabling these safety features on public roads if their technology is superior. If their technology is less safe, and in this case it appears to be that it is, they shouldn't be driving around on public roads. Keep it to test tracks. Volvo's system would have either stopped or slowed enough to make this crash non-fatal. That's worth pointing out. There is no point in Uber testing technology that is worse than what you can buy any number of cars today.
- FussyZeus 9y ago...playing Devil's Advocate, as GP said, it's common practice when testing any self driving tech to disable these features on the cars in question so your automated system isn't fighting with another set of automated systems trying to control the same vehicle. That sounds far more unsafe and unpredictable than just disabling the built in ones. Personally I think it's naive and unreasonable to think we're going to get totally safe and rock solid self-driving tech without at least a few coffins worth of people who gave their lives to show us how it could go wrong. There isn't one safety system in existence that wasn't built on top of a pile of corpses.
- polack 9y agoI've had a car with automatic breaking system for a while now and my experience is that they do not kick in easily. Had to test it with a friend that throw an inflatable mattress in front of the car to make it react. It should not be too hard to make these systems co-exist. If one of the systems says break then the car will break. The few times the car breaks when it really didn't need to should be an acceptable side-effect during testing.
- mannykannot 9y agoOnly yesterday there was this discussion [1], in which the general consensus was that the safe way to do things is to put the 'smart' goal-oriented driving stuff on top of the 'dumber-but-reliable' technology to handle specific dangers. It is the latter that has produced most of the statistical evidence justifying the claim that automation makes us safer. I am very skeptical of your claim that this is a less safe way of doing things. There is, in fact, communication between the two systems: the safety system 'says' to the driving system, "no you don't", and that should not be up for negotiation any more than it is when a human is driving. As was pointed out in the other discussion, if the driving system cannot find a way to work within the restrictions of the safety systems, it is doing it wrong. [1] https://news.ycombinator.com/item?id=16681611 https://news.ycombinator.com/item?id=16681611
- dharmab 9y agoYou could also record the decisions of the self-driving system and replay the scenario in a simulation environment (either virtual or closed course). Keep the stock safety features on for public roads and still get useful data.
- thrownblown 9y agoIsn't this how the human control systems are? Without reading the referenced convo, I can see many parallels. Like we have reflexes that we don't have full control over, that often don't even interact with our brain but rather are initiated from the spinal cord, which I would classify as dumb in this instance. And layered on top are emotional responses, memories and finally consiousness as, arguably, the control layer.
- kelnos 9y agoHaving a dumber system underneath the smarter system is a great idea. I'm not arguing against that. I'm arguing against having two separate systems that are unrelated and are built (likely) without a deep understanding of how the other behaves. Uber themselves should have a dumber system built underneath the smarter system (and maybe they do!); that way they understand all the variables that govern the operation of both and can be reasonably sure that the two systems won't fight each other. They can also build an interlock that causes the smarter system to completely disengage if the dumber system has to take over. I of course don't know this for a fact, but I wouldn't be surprised if Uber doesn't and can't have a complete understanding of how the Volvo system works. Disabling it makes sense. Again, they should replace it with their own dumber system, but that's a separate issue.
- amluto 9y ago> Even if communication between the two were possible, I'd still imagine that just having a single system operating the care would be much more desirable. If I had a Volvo XC90, I would want to operate it with two systems: the collision avoidance features plus an advanced system of muscles, eyeballs and brain. The whole point of collision avoidance is to operate in conjunction with a driver. Self-driving should be more or less just another driver.
- tzakrajs 9y agoAren't humans glorified self-driving systems? The second system is a backup for when the human makes a terrible mistake, similar to how this could've been a backup for the computerized self-driving system.
- fixermark 9y agoIt reads like a story press-released by Volvo to make sure Volvo is CYA'd in this in the public eye (lest uninformed consumers find themselves wondering how good Volvo's collision avoidance is if it couldn't stop that Volvo-branded car from hitting that woman in Arizona).