4 ms·

TBH, since it's a training based system it's "just" a matter of making sure the training set is large enough, including the situations you mentioned (assuming t

TBH, since it's a training based system it's "just" a matter of making sure the training set is large enough, including the situations you mentioned (assuming the training method is robust, generalizes well, etc). I would love someone knowledgeable to give an estimate, but I would guess you need at least a handful (10+?1000+?) of examples of each edge case (involving bicycles, pedestrians, weird road designs, street signs, and so on) -- and there are many of them I suspect (at least 100s?). Estimating you'd take about 1 hour between experiencing a tricky scenario while driving around, this should put the number of hours at something like 100,000+ -- not easy to come up with by himself (that's about 50+ years of driving 6 hours a day). Mobileye is doing something interesting by curating the reliable parts of the dataset (e.g. they have curated databases of traffic signs for each region) -- again not something you could do own your own, and seemingly archaic (hence GeoHot's criticism), but if you can afford it can speed up the training significantly. Tesla is a massive resource here because they already have a huge fleet of internet connected cars proving enough data to fill the aforementioned training set in a matter of days or months: let's estimate their fleet at 40,000 cars -- then they could fill that minimum dataset in less than a day, and in a month they might have a 100x safety margin. Of course, there's a big technical problem of relaying all that video (maybe they just relay prediction failures), but the data is there. Another fundamental problem with exclusively hands-off training (and little optimal control theory, etc) is picking up bad habits from drivers -- even the best algorithms will have a hard time and be only about as good as a good driver in each scenario, in the best case -- since the training data is acting as a ground truth.

vonmoltke 11y ago
> I would love someone knowledgeable to give an estimate, but I would guess you need at least a handful (10+?1000+?) of examples of each edge case (involving bicycles, pedestrians, weird road designs, street signs, and so on) -- and there are many of them I suspect (at least 100s?). It depends on what sensors are in use and how the environment affects them. I can't get into much detail unfortunately, but I have seen radar systems that use naive Bayes classifiers for target detection and classification. Those systems required large numbers of examples across a large, multi-dimensional space to work effectively. Target detection and identification is a trivial task compared to what the control system of an autonomous vehicle needs to handle.
1024core 11y ago
> I would love someone knowledgeable to give an estimate, but I would guess you need at least a handful (10+?1000+?) of examples of each edge case (involving bicycles, pedestrians, weird road designs, street signs, and so on) The problem is: there are new edgecases born every day. Consider, for example, an accident where the cops have set up flares. How often do you come across one of those? Very rarely, I imagine. And even if you did come across it in your training set: how does the ML know that you are following the cops' signals, and not just randomly switching lanes? That the flares are a critical signal?
- darkmighty 11y ago
  Good point, but if you consider the Tesla dataset... it's formidable. Every day they could collect data enough for ~55 years of driving a lot every day. Even if you never encountered this case, if it happens at all it's likely to be seen many times (probably 100+ in a few months) in that dataset. After driving cars have gone mainstream, this may start to be seen as a design problem by traffic agencies: they might standardize ways to deal with traffic a little more. Ultimately as long as the cars driving autonomously is small enough and procedures change slowly enough you should be able to continously update the driving system. But let me reinforce that a pure learning approach even with very large datasets may not be efficient as one would like -- the curation of signs is a good idea, and manually reviewing accidents and near misses (a highly human-intensive task) and perhaps flagging bad driving behavior (probably after some outlier screening, which can be good or bad) will be important to get it really good with the training-intensive approach (and not the top down optimal path planning and control approach). EDIT: Mobileye CEO discusses some interesting design issues and manual validation (and shows they have lots of data, good sign) https://www.youtube.com/watch?v=kp3ik5f3-2c&feature=youtu.be&t=7m51s https://www.youtube.com/watch?v=kp3ik5f3-2c&feature=youtu.be...
LoSboccacc 11y ago
what if a driver does a mistake, like running a red, and doesn't get a ticket? who validates all this data? attaching a dnn to a driver as a training set is a pipe dream, for now. maybe after we understand how our brain perceives time and build models of future outcomes, we could apply it to build better nn. For now, nn are just best used as classifiers in a controlled environment, not from an environment with unpredictable states. and especially not in an environment with adversaries http://spectrum.ieee.org/cars-that-think/transportation/self-driving/researcher-hacks-selfdriving-car-sensors http://spectrum.ieee.org/cars-that-think/transportation/self...
- darkmighty 11y ago
  The vulnerability to sensor error (adversarial or not) is certainly not exclusive to nn based approaches. I commented on the validation problem in the comment above and in another below, and one way to deal with it is simply manual validation (mainly for false positive elimination). Indeed this approach with dnns is already being employed by Mobileye, so I don't think it's a pipe dream. Sensor failure or well characterized adversarial inputs are actually really easy to deal with -- they are very easy to simulate with a given dataset and self-validate using traditional techniques -- simply make one or more cameras fail (or receive spurious sigs) and verify the output. It's a good point that probably all autonomous cars will need a contingency plan (probably human intervention and/or blind emergency stops) with non-zero probability -- even if you have a redundant network of cameras around your vehicle a critical number can and will occasionally fail (when you look at the fleet sizes that will be dealt with).

vonmoltke 11y ago

> I would love someone knowledgeable to give an estimate, but I would guess you need at least a handful (10+?1000+?) of examples of each edge case (involving bicycles, pedestrians, weird road designs, street signs, and so on) -- and there are many of them I suspect (at least 100s?). It depends on what sensors are in use and how the environment affects them. I can't get into much detail unfortunately, but I have seen radar systems that use naive Bayes classifiers for target detection and classification. Those systems required large numbers of examples across a large, multi-dimensional space to work effectively. Target detection and identification is a trivial task compared to what the control system of an autonomous vehicle needs to handle.

1024core 11y ago

> I would love someone knowledgeable to give an estimate, but I would guess you need at least a handful (10+?1000+?) of examples of each edge case (involving bicycles, pedestrians, weird road designs, street signs, and so on) The problem is: there are new edgecases born every day. Consider, for example, an accident where the cops have set up flares. How often do you come across one of those? Very rarely, I imagine. And even if you did come across it in your training set: how does the ML know that you are following the cops' signals, and not just randomly switching lanes? That the flares are a critical signal?

darkmighty 11y ago
Good point, but if you consider the Tesla dataset... it's formidable. Every day they could collect data enough for ~55 years of driving a lot every day. Even if you never encountered this case, if it happens at all it's likely to be seen many times (probably 100+ in a few months) in that dataset. After driving cars have gone mainstream, this may start to be seen as a design problem by traffic agencies: they might standardize ways to deal with traffic a little more. Ultimately as long as the cars driving autonomously is small enough and procedures change slowly enough you should be able to continously update the driving system. But let me reinforce that a pure learning approach even with very large datasets may not be efficient as one would like -- the curation of signs is a good idea, and manually reviewing accidents and near misses (a highly human-intensive task) and perhaps flagging bad driving behavior (probably after some outlier screening, which can be good or bad) will be important to get it really good with the training-intensive approach (and not the top down optimal path planning and control approach). EDIT: Mobileye CEO discusses some interesting design issues and manual validation (and shows they have lots of data, good sign) https://www.youtube.com/watch?v=kp3ik5f3-2c&feature=youtu.be&t=7m51s https://www.youtube.com/watch?v=kp3ik5f3-2c&feature=youtu.be...

LoSboccacc 11y ago

what if a driver does a mistake, like running a red, and doesn't get a ticket? who validates all this data? attaching a dnn to a driver as a training set is a pipe dream, for now. maybe after we understand how our brain perceives time and build models of future outcomes, we could apply it to build better nn. For now, nn are just best used as classifiers in a controlled environment, not from an environment with unpredictable states. and especially not in an environment with adversaries http://spectrum.ieee.org/cars-that-think/transportation/self-driving/researcher-hacks-selfdriving-car-sensors http://spectrum.ieee.org/cars-that-think/transportation/self...

darkmighty 11y ago
The vulnerability to sensor error (adversarial or not) is certainly not exclusive to nn based approaches. I commented on the validation problem in the comment above and in another below, and one way to deal with it is simply manual validation (mainly for false positive elimination). Indeed this approach with dnns is already being employed by Mobileye, so I don't think it's a pipe dream. Sensor failure or well characterized adversarial inputs are actually really easy to deal with -- they are very easy to simulate with a given dataset and self-validate using traditional techniques -- simply make one or more cameras fail (or receive spurious sigs) and verify the output. It's a good point that probably all autonomous cars will need a contingency plan (probably human intervention and/or blind emergency stops) with non-zero probability -- even if you have a redundant network of cameras around your vehicle a critical number can and will occasionally fail (when you look at the fleet sizes that will be dealt with).