16 ms·
Uber Neglected Simulation Testing on Self-Driving Cars, Insiders Say
- cabalamat 8y agoMove fast and break people.
- hannasanarion 8y agoThe Uber crash was not a failure of the AI, it was a failure of the UX. The car saw the obstacle and detected a collision, but there was no warning system to alert the driver about it. Meanwhile, the driver was looking at a laptop and taking notes on the system logs (which they were told to do), not watching the road.
- cozzyd 8y agoThe car saw the obstacle and detected a collision, but did nothing, presumably due to the high false positive rate of such things.
- aetherson 8y agoYeah, I feel like your parent poster is ignoring the implications of the high false positive rate. Like, let's say I had a grammar check. But the grammar check was bad. It flags every other sentence as needing a grammar change, anything other than a very simple "subject-verb-object" sentence structure looks wrong to it. For every correct grammar problem it identifies, it finds 20 incorrect grammar problems. As a result, I turn off the UI for the grammar check. Then I write a document that has a grammar problem. It seems stupid to say that this is either a failure of my use of the software, or a UI failure of the software. The problem is that the fucking software doesn't work, specifically that it fails in a way that makes it too "noisy" to use. And sure, if my job is to catch grammar errors, then it's also my fault. I'm supposed to be backing up the system. But we also know that I won't be perfect and will have failures of attention sometimes. But regardless of that, the grammar checker just didn't work. And, in this analogy, bad grammar kills people.
- voidmain 8y agoIn this case, as I understand it, they also disabled the emergency braking feature built into the Volvo. Which wouldn't be obviously crazy if they had better, adequately tested emergency braking integrated with the "autonomy" system. But to disable the existing safety feature in favor of their own system, and also disable their own system due to a high false positive rate, seems negligent on the face of it.
- joejerryronnie 8y agoAnd you have to correct the bad grammer in 0.5 seconds.
- hannasanarion 8y agoThe danger of false positives does not justify having no mechanism to alert the driver. The system is relying on the driver to take over in exceptional circumstances, but gives no warning when an exception occurs.
- _jal 8y agoIf you focus on the crash itself, yes, this semi-autonomous murderbot failed to alert the human. But the failure that killed Elaine Herzberg happened well before then, when the decision to disable the emergency brakes was made. It appears they disabled other safeties as well. I don't disagree that playing a little loose with allowing new tech makes sense. We need to understand what we don't understand about them, accidents of course will happen, etc. But this looks a lot like foreseeable negligent homicide to me.
- jacquesm 8y agoI'd argue it was a corporate culture failure. A company that can put a device like that on the road is failing on many levels, that there is a contributory issue with the AI or the UX has nothing to do with the root cause.
- w_t_payne 8y agoYou probably have something here. Culture is sooo important.
- cozzyd 8y agoI guess this is both astonishing and completely unsurprising at the same time. How can one hope to gather decent statistics in exceptional situations without a simulation?
- w_t_payne 8y agoA big problem is that the software engineers and algorithms engineers are pressed to deliver demonstrations in very short timescales. This precludes the necessary conversations about test interfaces, for example, making the task of the simulation team closer to one of reverse-engineering than of simulator integration.
- IncRnd 8y agoAnd people getting run over is the result of those software decisions by management, software engineers, and alg. engineers. Don't just be a cog in the machine, doing what one is told to do. Write software in the ways that are known to be correct and responsible.
- deleted 8y ago[deleted]
- w_t_payne 8y agoTrue. Sadly, sometimes this means saying goodbye to your colleagues and friends, and looking for new adventures in an organisation that is capable of acting in a more constructive and mature manner.
- _jal 8y agoThis is important. I hope engineers working autonomous cars think long and hard about whether they want to be on a team that, it looks to me like, negligently killed someone by choice. The only difference between disabling the emergency braking mechanism and, say, intentionally disabling safeties on industrial machinery is the novelty. I think it was pretty obvious that something like this would happen, and that's why so many people have been making noise about it. Now that we know for a fact that Uber is willing to make this choice, there's (a) no excuse for not coming down on them for it, (b) no excuse for Uber not completely overhauling their safety protocols with the help of an external ethics and legal advisors, and (c) no excuse for engineers working on this for not at least seriously thinking about their own ethics and possible moral culpability. Some people want to focus on the future. Self driving cars will be great! Gotta break some eggs to make omelets! Fine. But those who are killed before that bright future deserve some accountability, those who don't want to be killed deserve some respect, and the makers of said omelets can bloody well be expected to be held responsible for their actions.
- deleted 8y ago[deleted]
- duncanawoods 8y agoI don't understand why Google/Uber are live testing with lethal SUVs instead of polystyrene golf-carts that could only hurt someone if they tried to eat it. We could be getting acclimatised to near disposable pizza delivery carts earning the algorithm enough trust for 2 tons of metal and a passenger.
- sdhgaiojfsa 8y agoI can think of half a dozen likely reasons. I'm not sure which one it is, or if all of them could alone be fatal to your idea. - Such vehicles would not be street legal. - They would not be safe for passengers. - They would be unable to carry the necessary payloads. - Most of the weight of a car is in the chassis anyway, not the body. Edit: at any rate, Google did do something like what you described at an earlier stage of testing. https://www.youtube.com/watch?v=HJnjuhBIhwc https://www.youtube.com/watch?v=HJnjuhBIhwc They seem to be past that point, though.
- ams6110 8y agoIn a modern car, the chassis and body are one and the same (unibody construction). Some cars do use plastic panels for non-structural parts. I would think another big reason though is that the handling and response of such a car would be very different from a real car. Stopping distances, acceleration, cornering, wind behavior, would all be different.
- jerf 8y agoI can easily believe in code that adjusts to the underlying vehicle's differences. It seems to me the real problem would be that A: You need the car to be on the real streets, so at a minimum, it must be street legal. B: You need it to be on all the streets, including highways and such, so we're at least up to a Smart car level of body. From there it really isn't that great a leap to a full sedan chassis. Would one of those little cars even have room for all the requisite sensors? Even if they did, sedans are just so much more useful. So it seems like the minimum practical car to test with is a sedan, and that's enough to kill people. Even the little Smart cars are plenty enough to kill people.
- rasmi 8y agoFor comparison, here is a look at Waymo's simulation operations: https://www.theatlantic.com/technology/archive/2017/08/inside-waymos-secret-testing-and-simulation-facilities/537648/ https://www.theatlantic.com/technology/archive/2017/08/insid...
- dclowd9901 8y ago> Being in the car a lot, I can feel what the car is doing—it sounds weird, but—with my butt He's describing, incidentally, how professional drivers drive.
- deleted 8y ago[deleted]
- hangonhn 8y agoCan you expand on that a bit more?
- Tuxer 8y agoAs you become "good" at driving in harder conditions, you'll use your butt (which is really how you're physically connected to the vehicle) to figure out the traction state of the car, to know if the car is sliding, if it's close to losing the rear end or the front end, if it's pitching down or up, and so on. This is how you know which corrections to make to the vehicle to keep it going fast without losing control. Pilots of small planes do the same thing. After a while you can just feel small accelerations and change of directions without needing to check some of the instruments.
- QuercusMax 8y agoHuh, fascinating. I remember reading a Star Wars novel ages ago in which they said that one of the things that made Luke Skywalker a good pilot was that he slightly detuned his craft's inertial compensators (or whatever the Star Wars version is) so he could feel how the craft responded. Didn't realize that was actually based on real life driving/flying!
- srcmap 8y agoIt should be mandatory for all self-driving car (system/software) vendors to publish their test plans and all test logs for public to review before the car/software are allowed to be on the public road. It likely allows everyone to learn from all the mistakes and improve the process for the whole industry - similar to the passenger airplane safety record improvements learned from every single accidents/crashes.
- mrep 8y agoThe potential problem with that is it reduces everyone's incentives to do it in the first place because they can just leech off of their competitors.
- w_t_payne 8y agoThey do that anyway, to an extent.
- blevin 8y agoPerhaps we will end up with a standardized validation simulation model that autonomous vehicles must pass for licensing. The test suite could grow as more anomalous scenarios are analyzed (ex: acceleration into gore point; white-painted trailer truck turning across divided highway; debris on highway), and it would ensure that any new vehicle on the market or OTA update would meet a minimum of safety. Manufacturers could compete on test suite coverage and acceleration curve stats, and market using rendered video of simulation results showing superhuman skill.
- w_t_payne 8y agoI hope there is some level of cooperation. At the very least we should have a standard set of crash or near-miss scenarios so that we can collectively learn lessons when things go wrong. Particularly when people have already paid the price with their lives.
- w_t_payne 8y agoThat is not a bad idea ... although if they published the ones that they have now, it might give people some pause.
- beat 8y agoGiven time to market constraints and the potential value of having widely adopted autonomous vehicle software, it's not all surprising that vendors will take experimental shortcuts. And it's not at all surprising that sometimes, those shortcuts will result in fatalities. That being said, I still want this tech on the road ASAP, not because it's perfect, but because human drivers are So. Fucking. Awful. They kill thousands of people a year due to negligence, incompetence, and occasional malice. Plus, the additional sensors available and the software response time mean autonomous drivers can outperform even sober, attentive, skilled human drivers in most situations. A couple of weeks ago, I saw someone talking scary stuff about what someone here called "semi-autonomous murderbots" in a FB post that appeared on my feed immediately after several posts about a local accident, where a bicyclist was killed by a road-raging driver, who then hid/destroyed his car (the murder weapon still has not been found, even though his trial is now over). If that jackass had a self-driving vehicle, the bicyclist would still be alive today. So are big corporations going to do kind of a careless job testing live vehicles in a rush to market? Yes. Will people get hurt? Yes. Is it worth it? Yes.
- frgtpsswrdlame 8y ago>That being said, I still want this tech on the road ASAP, not because it's perfect, but because human drivers are So. Fucking. Awful. This narrative is so pervasive and I must admit a bit strange to me. Millions and millions of people strap themselves into thousands of pounds of metal and then drive 80 down the highway while texting and changing the radio and telling your kids to please stop fighting surrounded by other people doing the exact same thing. And we do that every single day and almost all of us are fine and will never have a 'major' accident. Humans are pretty fucking great at driving. Besides to say that human are bad (or good) at driving would require some comparison, we're bad at it compared to what? As far as we know humans are the best drivers in universe. There's no evidence that self-driving is currently better than human drivers and there's no evidence that self-driving cars will soon be better than human drivers and there's no evidence that if we allow tons of self-driving cars on the roads then they will be better than humans. It's just blind hope, it really makes me want a word for whatever the opposite of a luddite is, instead of a belief that we can destroy machines to benefit people we believe we can destroy people to benefit machines. It's all twisted up. If you want to be an Uber or Tesla crash test dummy and walk/bike across a test track while they see if their vehicle can avoid you then by all means go for it. But don't just foist that upon average people.
- rossdavidh 8y agoThe right question is not "why did Uber do this?", but rather, "what was it you thought would stop them from doing this?" Current answer: nothing. But hopefully, the attitude of governments towards letting a company like Uber test fast-moving heavy machinery on their citizenry, is changing. Different industries have different attitudes towards innovation and risk, and there are good reasons for that difference.
- w_t_payne 8y agoI think that the automotive industry could take a leaf from the aerospace industry. A lot of aerospace money has been spent learning how to do things safely. It would be a shame to spend the same money all over again, re-learning the exact same lessons in automotive.
- DmenshunlAnlsis 8y agoIt’s not about learning, it’s about allowing. Some companies like Uber and Tesla will do whatever they can get away with, for advantage in the market and ultimately money. The aerospace industry is safe because it is massively regulated.
- w_t_payne 8y agoA lot of the carriers and manufacturers already have a very strong safety culture. The regulator spends most of its time working with smaller companies.
- DmenshunlAnlsis 8y agoThat safety culture wasn’t born de novo, it was legislated into being after years of fighting with them. Remember Ralph Nader? The companies that now have a “strong safety culture” were the same that fought against seatbelts for Christ’s sake. Airlines had to experience a decade+ of terror attacks from rank amateurs, Pan Am had to die before basic security screening was accepted.
- 8y ago
- pmarreck 8y agoOf all the things that managers neglect to ensure rigorous testing of, self-driving cars should be the worst possible offender
- w_t_payne 8y agoI would have hoped that ISO 26262 would help to prevent this sort of thing from happening. Maybe it is an audit quality issue?
- pmarreck 8y agoThere is also likely difficulty in testing an ML black-box. When I took the ML class (Stanford's, online), they actually recommended setting aside some percentage of the training data as test data, but this is (IMHO) sub-par because it doesn't eliminate the possibility of things like overfitting
- kerng 8y agoTesting in Production. It still boggles my mind that that's the norm in software engineering these days. Interesting how in this case it seems "inappropriate" in real life, and the article fails to make the connection how normal this is in the software world nowadays unfortunately.
- refurb 8y agoNot sure if this is possible, but couldn't they capture all the LIDAR inputs from a car being driven by a person, then feed that into the software to see if it actually recognizing objects correctly? Or even do it on a closed track. That way you could throw all sorts of things at it like kids, bikes, garbage cans, whatever. Should be able to see what works and what doesn't work pretty quickly. EDIT: Never mind! That's what Waymo does in the article another person posted. https://www.theatlantic.com/technology/archive/2017/08/inside-waymos-secret-testing-and-simulation-facilities/537648/ https://www.theatlantic.com/technology/archive/2017/08/insid...
- Klathmon 8y agoWhen it comes to "closed track" and other "simulation" style things, you need to be very careful when ML is involved. I don't know if Uber/Waymo/Others have "solved" this problem, but ML systems have a tendency to "learn" the wrong thing. You could run millions of simulations (both virtual or on a closed course) with "test obstacles", but the second the car encounters one a child with red balloons in the early morning on a foggy day it might not know what to do with it because that exact scenario wasn't in any simulations.
- caboteria 8y agoThey'd be using the simulator for validation, not training.
- refurb 8y agoExactly. Throw something at it the system should stop for (e.g. a child). If it doesn't, go back to the drawing board and figure out why. Agreed the test track is not ideal, that's why I was wondering if they could throw real recorded data at it. No risk to the public, but puts the system through real-world trials.
- locacorten 8y agoIn my opinion, this is criminal behavior and Uber's senior leadership should go to jail.
- wpietri 8y ago100% agreed. Somehow large corporations get away with a diffusion of responsibility. Enough people are involved that it's hard to blame one person. But certainly for issues resulting in death, I don't think responsibility should be diluted. If 10 different executives knew or should have known that their decision had life-critical implications, then I think all of them should face the appropriate penalty. (Here, I presume it's for involuntary manslaughter due to criminal negligence.) And I think the same penalty should apply to developers who knew or should have known that their actions put lives at risk.
- ConceitedCode 8y agoI never understood why the starting point for these projects was not simply to drive a car in GTA V (or some other game) around the city. It would test pedestrian detection, street signs, traffic lights, merging, emergency vehicles, etc... While certainly not perfect, it would seems like a good starting point in a controlled environment that is already modeled on the real world.
- glup 8y agoIsn't it? I was under the impression that this was the first step was simulators like CARLA (http://carla.org/ http://carla.org/) before real-world testing.
- ConceitedCode 8y agoThanks for the link! That is a fascinating project. I was not considering the different sensor inputs that would not work on an existing game.
- crooked-v 8y ago"Move fast and kill people" is the new motto to use, I guess.
- qasar 8y agoQasar here (x-YC founder and partner). So im building a simulation company. Not commenting on Uber specifically but the reason lots of companies don't have great efforts is simulation is actually really hard. It’s not building a web application (tested methodologies, established best practices, lots of experienced engineers in the space...etc). You can get a lot done (and get wonderful demos) through research vehicles. The issue with sim is similar to the issues with AVs as a whole - getting to 90% is much easier than getting to 99.9999%. Also, there is this weird thing in AV (and specifically in-house sim) where pointy-haired managers think "if we put 30 SWEs on the problem - we'll make progress!". While raw eng horsepower matters in some software problems, when you are on the line between research and engineering, more SWEs might actually make getting a robust product up harder. Another thing we've seen is in-house efforts overfit solutions. They don't really have a wide viewpoint (nature of how secretive the space is), and therefore go down paths which are just plain wrong but no one is there to challenge them.
- w_t_payne 8y agoVery interesting perspective. I always assumed people would integrate an external simulator like CarMaker rather than build their own. I'd be exceedingly interested in hearing about your product when it comes available (although my interest is in simulation for maritime autonomy rather than automotive).
- qasar 8y agoWe're super familiar with CarMaker -- great for dynamics and some level 2 stuff but there are some big difference between autonomy levels (from simulation perspective at least). At higher levels, autonomy algo development is more like modern software development. CarMarker is a fantastic product (along with CarSim, dSpace ASM...etc) but they were built for a different use cases. A very crude analogy would be tools for J2MEE vs modern tools for iOS/Android. Our product is available, just not very public about our more detailed approaches because that is really where our IP is. Feel free to reach out if you want to chat. Obv, i'm super interested in the topic.
- solatic 8y ago
- sumit036 8y agoIf you want to know about more about self driving cars then you should click here <a href="https://skillmaking.com/waymo-self-driving-cars-google-self-driving-driverless-car/">waymo https://skillmaking.com/waymo-self-driving-cars-google-self-... self driving cars</a>