9 ms·
End to End Learning for Self-Driving Cars [pdf]
- birdmanjeremy 10y agoThis is really interesting, but I'd want autonomous cars to be better than humans at driving, not to emulate them.
- hedgehog 10y agoThe neural net that accurately predicts human control inputs also extracts the relevant features you would want to build a more principled autopilot. For example you could take one of these nets and then for build a speed limit sign locator on top of it without training a whole net for that from scratch. Using that you could then hard code the rules for obeying the speed limit into a more traditional planner.
- rdabane 10y agoYeah,make sense. But this is the first example I've seen where the network is trained only with driver's input. Has anyone seen this kind of approach before?
- cryptoz 10y agoThere are real potential advantages to training networks using human behaviors. For example, the person-to-person communication that takes place while driving might be better understood by machines this way. - Was that an obscene gesture or a thank you wave? - Did the other driver suggest I go forward or tell me to stop? - etc
- rdabane 10y agoWondering if they have to create near collision scenarios in order to train the network :-)
- anonymfus 10y agoThis technology is much more universal than just autonomous cars.
- andrewchambers 10y agoThey need some sort of prefilter to remove any bad driving habits before training begins I suppose.
- jonny_eh 10y agoI imagine with enough training data from enough drivers that bad habits will disappear as noise.
- msandford 10y agoDepends on the "bad habit". A huge number of people speed. How's that going to disappear as noise?
- cwe 10y agoI'd like to believe that would be the case, but I live in Seattle.
- andrewchambers 10y agowouldn't it just converge on an average driver? not even a good one.
- johntb86 10y agoFrom the paper: To train a CNN to do lane following we only select data where the driver was staying in a lane and discard the rest.
- jonny_eh 10y agoEven if it was trained to drive like a human, it'd be better in many ways: It never gets drunk, doesn't get tired, doesn't get distracted, can drive old people (or anyone) around who shouldn't be driving anymore, etc.
- blazespin 10y agoYeah, but if a computer screwed up like a human we'd never hear the end of it.
- taf2 10y agoThis almost sounds like the line from terminator two. "It can't be bargained with. It can't be reasoned with It doesn't feel pity, or remorse, or fear. And it absolutely will not stop, ..."
- smackmybishop 10y agoThat's the first Terminator. Terminator 2 would be: It would never leave him, and it would never hurt him, never shout at him, or get drunk and hit him, or say it was too busy to spend time with him. It would always be there. And it would die, to protect him.
- jfoutz 10y agoI couldn't find the article, but i remember a project learning from remote control plane pilots. People mess up all the time. It's not even drunk or tired, there's just errors every so often, it's almost statistical. The little errors people make wash out as noise.
- blazespin 10y agoGood point. But there are a lot of things like self driving cars where emulating would be a quantum leap forward...
- elihu 10y agoI've read that one of the problems with Google's self-driving cars has been that other cars tend to run into them because the self-driving cars drive extremely conservatively and violate other driver's expectations of how a typical California driver is expected to behave. I think this sort of thing is something developers are going to have to find ways of dealing with; a car can be technically driving in a safe, legal way but if it's too different from how a human would drive, they are going to be a safety hazard. Of course, standard driving behavior varies dramatically from place to place. For instance, in the United States, everyone is expected to get out of the way of whichever car has the right-of-way in that situation. In Indonesia, the car that has the right-of-way is expected to slow down, stop, or move over to accommodate other cars that do things like pull out in front of them in an intersection or pass on a two-lane road with oncoming traffic. A self-driving car in Jakarta would need to be trained very differently than a self-driving car in Seattle or Paris. Not just because the traffic laws are different, but because drivers have very different expectations about what is normal behavior.
- blazespin 10y agoSelf driving cars must do significantly better safety wise otherwise the adoption will be hampered. Plus, the whole point here I think is to save lives. Google self driving cars arent really a hazard they're more just very annoying because they are overly cautious.
- mechanon05 10y agoI feel like people just assume that they are annoying to drive around, but very few people actually have experience driving around google's cars. The times that I'm around them (a few mornings a week), they are never in any way weird or annoying. In fact, they are extremely predictable, and therefor, if anything, less annoying to drive around.
- mikeyouse 10y agoI agree -- they never encroach in your lane, they signal with plenty of room to spare, they don't threaten to pull out in front of you, in many different ways, they're preferable to human (Californian) drivers.
- dpflan 10y agoI think the trick is not to mix autonomous and non-autonomous. Cities delegate autonomous only zones where no non-autonomous cars can go; this creates a transportation circulatory system and safe experimental zone which can expand - which can include just making the area of city only for autonomous cars larger or gradual commingling with non-autonomous cars. Or again partial autonomy of cars on highways / more predictable driving scenarios, like we are already seeing.
- sammyo 10y agoBut this is just wrong. Virtually the only accidents were people rear ending a stopped car at a red light. The current google cars are probably improving the general driving safety.
- robbrown451 10y agoThat seems like the trick to making sure they are never adopted in a big way. I mean sure, that's an easy problem to solve but in that case why use cars at all and not people movers or the like?
- marcosdumay 10y agoNo problem in it driving like the best human drivers around.
- sqeaky 10y agoNvidia is one of the main reasons my computers, I do not want this true of my cars as well. Edit - Is a joke really worth this many downvotes? I mean who hasn't had occasional trouble with Nvidia drivers and games?
- kcbanner 10y ago"Nvidia is one of the main reasons my computers, I do not want this true of my cars as well." If this was a joke I think the grammatical error led to confusion?
- kajecounterhack 10y agoCool trivia: the building is in Holmdel, NJ and is really nice but sat abandoned for years. Companies can rent coworking space now, they're rebranding as "Bell Works" (http://bell.works http://bell.works) https://en.wikipedia.org/wiki/Bell_Labs_Holmdel_Complex https://en.wikipedia.org/wiki/Bell_Labs_Holmdel_Complex
- dang 10y agoUrl changed from https://blogs.nvidia.com/blog/2016/05/06/self-driving-cars-3/ https://blogs.nvidia.com/blog/2016/05/06/self-driving-cars-3... to the paper it points to.
- rdabane 10y agoThanks.
- khalilravanna 10y agoAn Nvidia self-driving car. Impressive...but can it run Crysis? In all seriousness though I'm really enjoying the number of corps exploring the space of self-driving cars. It can only make the reality of a road full of autonomous cars come all the sooner. (Though with the rate at which I see Google's self driving cars around Austin you'd think they were already out for public consumption.)
- rdabane 10y agoIts amazing how the characteristics defined in (https://en.wikipedia.org/wiki/Contextual_learning https://en.wikipedia.org/wiki/Contextual_learning) apply directly to this scenario. (Learning from a master )
- StavrosK 10y agoIt's mind-blowing to me that they did raw image mapping to steering angle, without any manual feature extraction whatsoever. This is just revolutionary.
- euyyn 10y agoOne problem with training a neural network end-to-end this way is that the system is susceptible to unpredictable glitches: The same principle that lets people trick a NN into [thinking a panda is a vulture](https://codewords.recurse.com/issues/five/why-do-neural-networks-think-a-panda-is-a-vulture https://codewords.recurse.com/issues/five/why-do-neural-netw...) can happen randomly just by differing lighting/shadow conditions, sun glare, or who knows. One can always train the network with more and more scenarios, but how do you know when to stop? How good is good enough in this regard?
- rawnlq 10y agoFor the "thinking a panda is a vulture" problem, don't humans fail in similar ways? The analogous examples for us are camouflage, optical illusions, logical fallacies, etc. It doesn't really have to be perfect as long as it doesn't fail in common scenarios.
- rdabane 10y agoIt should never fail since any failure could potentially create a fatal scenario. People usually accept fatalities because of human error but they won't accept death because of algorithmic failure.
- Consultant32452 10y agoI suspect that it won't take long for people to come to terms with it in the same way we now "accept" industrial accidents. "Accept" in this case simply means that the industry in question is allowed to continue doing business.
- kabouseng 10y agoThat's an unattainable high acceptance bar. A more reasonable one would be to have mass adoption of self driving cars as soon as self driving cars cause less accidents than human drivers.
- monk_e_boy 10y ago
- ChuckMcM 10y agoWow I guess having a self driving car is the new thing next we'll have self driving sli, can drive two cars if they are joined together with a proprietary cable :-) More seriously, this is amazing work, I am really impressed, I just wonder if we can't get something more topical like training lasers on mosquitoes or something. I feel like I did when 3D graphics was the new thing, every day it felt like there was a new advance, now the same tech is doing the same thing to machine learning.
- euyyn 10y agoAnother obvious problem with having the neural network map from vision to steering is that you can't make it take decisions based on what it hasn't seen yet, but will see. E.g. changing lanes because to reach your destination you'll have to make a right turn. The authors note that E2E learning makes for a better smaller system, on account of being free from human-imposed concepts. That's fine if that's your only goal. But I think it's essential for autonomous cars to be able to reason in terms of those human concepts.
- argonaut 10y agoThis doesn't describe the (very basic prototype) system in the paper, but there is no theoretical reason there cannot be a recurrent net that plans ahead (obviously developing such a system is extremely difficult)
- euyyn 10y agoWhat would you give it as an input, though? Real-time screen captures of Google Maps in navigation mode?
- brianchu 10y agoHere is a great, very accessible video where the CTO of MobileEye (which provides some of the components for Tesla's "autopilot") explains his views on the challenges of end-to-end learning for autonomous vehicles, and why it's preferable to decompose the problem instead (still using deep learning for the decomposed modules). https://www.youtube.com/watch?v=GCMXXXmxG-I https://www.youtube.com/watch?v=GCMXXXmxG-I I'm inclined to agree, especially because it helps in 1) providing diagnostic information (such as the great driving visualizations shown in the video), and 2) makes it easier to incorporate algorithms and sensors (like with Google's cars) as a redundancy in case the neural network hits a crazy edge case.
- drewm1980 10y agoEngineers are going to break down the problem into many subsystems and test the heck out of them. Maybe the system can still be globally optimized, though, as long as individual subsystems are still verifiably correctly trained. i.e. lane detection, pedestrian detection could share some of the same convolutional layers and still be tested separately. My personal prediction is that all of this 2D convolutional network stuff will be extended to 3D within a few years. The front-end will do a full 3D scene reconstruction from first principles, and then some sort of 3D features will be learned on ~that data.
- ilyaeck 10y agoWith just 72 hours of training data, this is an extremely impressive result, scientifically speaking. However, in terms of quality control/best practices in automotive, it's simply unacceptable to trust human lives with a monolithic black-box system. The reason to have modular building blocks in a production system is not necessarily better performance, but the must-have ability to test, troubleshoot, debug, fix and replace things by reducing the degrees of freedom - it's the ABC of engineering even in much less sensitive industries. So while N2N is going to be very useful for rapid prototyping, and perhaps setting the performance bar for other method, I doubt it will ever be used in production.
- EugeneOZ 10y agoCan be used as additional system for taking decisions.
- ksml 10y agoThis is a great point. Reminds me of how airplanes have redundant flight computers and compare the outputs of the computers to determine if one might be faulty. https://en.wikipedia.org/wiki/Fly-by-wire#Redundancy https://en.wikipedia.org/wiki/Fly-by-wire#Redundancy The output of different self driving models could be compared to handle more difficult driving situations -- I never thought of that.
- iamaaditya 10y agoThis reminds me of ensemble. Most machine learning techniques benefit in accuracy and precision by ensembling different methods. More different the methods are (variance) better improvement you get.
- Animats 10y agoThe machine learning, no model approach is kind of scary. It's likely to do the right thing most of the time, and something really bogus on rare occasions. There needs to be more than just a model trained from successful driving. Some kind of recognition of "this is bad" is needed.
- rdabane 10y agoYup, like a near collision scenario to teach it to drive off the road instead of head on collision ...
- infocollector 10y agoDoes anyone know which camera they were using?
- hokkos 10y agoSeems very similar to the GeoHot self driving car experiment.
- FriedPickles 10y agoIn case you missed it, the ninth reference is a good video of the system in action: https://drive.google.com/file/d/0B9raQzOpizn1TkRIa241ZnBEcjQ/view https://drive.google.com/file/d/0B9raQzOpizn1TkRIa241ZnBEcjQ...
- webaba 10y agoIn aeronautics, we were able to increase safety to an insane level by understanding the physics of the environment, formally proving and certifying algorithms, using Robust control theories that allow to formally deal with uncertainty. The power of mathematical modeling together with robust software testing and a limited/controlled use of learning algorithms is - in my opinion - likelier to bring safety to such systems rather than such an opaque use of CNNs. And I'm not even talking about human machine interaction or liability issue that would emerge from such extreme approaches (which node of the net or which training sample will be blamed?). I guess CNNs allow a lot of wannabe engineers to play with real world problems and dream that their 10 lines python code would match semantic models if fed with more training data, but I'm pretty sure Aircraft/Car manufacturers will/should not replace formally certified controls algorithms and redundant architectures built on the top of hundreds years of analytical results with a rack of NVIDIA GPUs.