8 ms·
An Unintuitive Take on Data Augmentation for Self-Driving Cars
- zaroth 8y agoForgive me hijackjng a self-driving ML discussion to ask a novice question on self-driving — “stateful” vs “stateless”, and how can state be used safely? Specifically I mean pre-determined saved/downloaded knowledge about the current route being driven, or your specific instantaneous location. Anything from annotated maps, to LIDAR scans. It seems like the paradox is that stateful algorithms can dramatically improve performance by factoring in data that real drivers benefit from tremendously, namely, familiarity with the road. However detecting when known state is violated due to some temporal change (road work, accident, natural disaster), and being able to shift back into “general flight” rules, seems in some ways to be even trickier than not using historical state in the first place. So you need an algorithm that can learn to drive better than a human, without having basically any contextual knowledge of the road it’s driving on. In other words the algorithm needs to be a better driver the first time it ever goes down a road than a human who has driven the road for years. As a human, familiarity with the road makes a tremendous difference for how I drive it. I generally drive the same roads 80% of the time (I.e. commuting) and knowing what to expect around each bend absolutely changes how I drive the road, even down to where the speed traps will be. Any ML driving solution that depends on super-fancy stateful pre-scans of the environment seem fundamentally flawed. If you can’t drive a road safely that isn’t pre-scanned in hidef LIDAR for instance, I don’t suppose you can safely drive that road on an arbitrary Monday. Maybe solutions like this were never even attempted, but certainly some amount of statefulness is inherent in some of the commercial solutions out there (Supercruise?) So what kind of state can you use safely? First thought was basics like speed limit, number of lanes, type of road surface, and your algorithm may have predictions of upcoming changes being constantly weighted against the current assessment of the present state. So maybe the fundamental rule is no hard-coded state that can’t be reliably detected in real-time to the point where at some point a real-time signal is able to over-ride the programmed state? But then it seems that inevitably you get to the point where your real-time classifiers are basically running the show anyway so are you back to - what good is a Map anyway? Would love interested but accessible readings on the subject, or maybe it’s so off base that it’s not really part of the discussion?
- vladislav 8y ago"But then it seems that inevitably you get to the point where your real-time classifiers are basically running the show anyway so are you back to - what good is a Map anyway?" Yep, pretty much. Robust real-time perception, for a wide variety of tasks that happen very rarely is one of the key bottlenecks.
- nine_k 8y agoI wonder how humans can benefit from pre-knowledge and be able to respond to the unexpected. Can this be modeled somehow?
- CamperBob2 8y agoThe question of statefulness reminds me of some not-immediately-intuitive observations from game development. Say you're working on potentially-visible-set computation for a first-person shooter. Where do you spend your optimization efforts? Do you put a lot of work into taking advantage of object coherence between frames, for instance? Chances are good that something that's visible in one frame will remain visible in the next, right? So if you start with that assumption you might think you could save some time by not bothering to test any surfaces on that object for visibility until some nominally-unrelated condition is met, such as the player turning rapidly in place. You can waste a lot of time thinking about optimizations like that, but at some point it'll occur to you that it's a giant waste of time to optimize for anything but the worst case, where visibility information computed during one frame is completely unusable in the next frame for whatever reason. Otherwise all that your clever frame-coherence hacks can ever do is speed up the rendering of the sorts of frames that weren't going to dominate the player's perception of the game's performance anyway. An engine that renders 95% of frames at 60 FPS and 5% at 43 FPS is going to look pretty terrible, so you're usually better off putting work into the slowest frames rather than wasting time looking for hacks and shortcuts that make the fast frames even faster. Likewise, yes, you can assume that the car will almost never be traveling backwards on the left side of the road or whatever, so the temptation to take advantage of that is going to be high. But the cases where that assumption breaks down will hurt the user's experience badly, possibly disastrously. So you're better off without relying too much on assumptions that contain phrases like "hardly ever" or "most of the time" or "typically."
- Animats 8y agoThat's scary. Using deep learning to recognize "obstruction" using cameras. Discriminate "car", "bike", "pedestrian", and "bus", sure. Recognize signs and traffic lights, fine. But "obstruction"? No. That's a geometry problem. If it's not flat road, it's an obstacle. Deep learning is for deciding what kind of obstacle. Because deep learning just isn't that good. It's going to be badly wrong a few percent of the time. "The car will almost never be on the left side of the road, and the cameras will never flip angles, so training on flipped data forces the network to overgeneralize to situations it will never see." What could possibly go wrong?
- gok 8y agoWhat aspect of "deep learning" are you concerned about? Would you prefer obstruction detection be done with shallow NNs? Or a linear classifier?
- Animats 8y agoWith something that measures depth and can profile the ground. Probably a LIDAR.
- deepnotderp 8y agoLidar can get into trouble with obscurants. Don't get me wrong, lidar is an amazing sensor, but not perfect
- KKKKkkkk1 8y agoIf DNNs are unreliable, why is it ok to use them to discriminate a car from a pedestrian? The planning that you do in order to avoid hitting a car is very different from what you need in order to avoid a pedestrian. If you erroneously treat a pedestrian as a car, you are very likely to cause a tragedy.
- icebraining 8y agoAs a non-driver, I'm curious, how different is the planning? Isn't it just a matter of size?
- w_t_payne 8y agoIt is interesting to compare this result to the results on domain randomization on synthetic data...
- YeGoblynQueenne 8y ago>> It’s worth noting that these augmentation tricks won’t work on datasets that include images from different camera types, at different angles and scales. No comment on why this is the case?
- wcoenen 8y agoI find it interesting that the project team had been implementing several "improvements" that actually worsened performance, and that it took an intern to figure this out. Or are these augmentations default-on options in deep learning frameworks?
- Piskvorrr 8y agoSays so in the article. Essentially "we do this because everybody else does, therefore it's good." Note the title "ML practitioners" - it's used akin to alchemy these days: if the stars are right, it works.
- currymj 8y agonot turned on by default but they are sort of a default choice for vision tasks. it may also be that they were useful on an earlier iteration of the model, and only became harmful later. nobody’s quite sure why anything works, though. hence comparisons to alchemy — we have lots of cool empirical results, and a highly elaborated set of theories and ideas trying to explain what works and what doesn’t, but in fact these theories often fail to make correct predictions. So the only way to make progress is constant blind experimentation.
- oldgradstudent 8y agoYou're right to put the "improvements" in scare quotes. He improved the performance in normal conditions, but he might have made the performance outside normal catastrophic. When you can't kill anyone more often than once in 100 million miles, this might not be the right tradeoff.
- Piskvorrr 8y ago"and the car will always be on the right side of the road (assuming US driving laws)." In other words, overfitting to USA, questionably usable anywhere else, definitely not in UK? Okay, this quote is worth preserving.