6 ms·

> It's not possible to engineer an autonomous system that never fails, but it is possible to engineer one in such a way that it never fails to detect that it ha

by dancek 9y ago

> It's not possible to engineer an autonomous system that never fails, but it is possible to engineer one in such a way that it never fails to detect that it has failed. This is a very good point: just like a human driver should slow down if they can't observe the road ahead well enough, an AI should slow down when it's not confident enough of its surroundings. This is probably very difficult to do, and I'm skeptical about your claim that an AI can be engineered to always detect its own failures. Further, I naively believe that Tesla is already doing a lot to detect conflicting inputs and other kinds of failures. Maybe being careful enough would prevent autopilot from working at all, so compromises have been made? I probably don't understand AIs well enough to understand how they can be engineered to (almost) always recognize their own failures. But if a simple explanation exists, I'd love to hear it.

DSingularity 9y ago
I feel that’s the problem. Techniques like SVM can provide reasonable definitions of confidence. Reinforcement Learning.. DNN... the mechanisms behind autonomous cars... are they able to do this??
- xiphias 9y ago
  Neural networks are just non-linear functions that give probabilities to classes, so yes, it can be definitely done (and is done). As Tesla's computer vision system is not as good as Waymo's, it has to choose between comfort and safety more often.
- mhermher 9y ago
  not an expert, but aren't the output nodes thresholded to make a decision? How far you are from the threshold might be interpretable as confidence possibly?
  - xapata 9y ago
    For a single perceptron, not for the network as a whole.
  - CardenB 9y ago
    Yep that's exactly what you would do. However, in practice, this usually is just not a good indicator of confidence. NNs are notoriously overconfident.
- gwern 9y ago
  > Reinforcement Learning.. DNN... the mechanisms behind autonomous cars... are they able to do this?? Sorta. There are ways to extract kinds of uncertainty and confidence from NNs: for example, Gal's dropout trick where you train with dropout and then at runtime you use an 'ensemble' of multiple dropout-ed versions of your model, and the set of predictions gives a quasi-Bayesian posterior distribution for the predictions. NNs can be trained directly via HMC for small NNs, and there are arguments that constant-learning-rate SGD 'really' implements Bayesian inference and an ensemble of checkpoints yields an approximation of the posterior, etc. You can also train RL NNs which have an action of shortcutting computation and kicking the problem out to an oracle in exchange for a penalty, which trains them to specialize and 'know what they don't know' so they choose to call the oracle when they're insufficiently sure (this can be done for computational savings if the main NN is a small fast simple one and the oracle is a much bigger slower NN, or for safety if you imagine the oracle is a human or some fallback mechanism like halting). I have some cites on these sorts of things in https://www.gwern.net/Tool-AI https://www.gwern.net/Tool-AI and you could also look at the relevant tags https://www.reddit.com/r/reinforcementlearning/search?q=flair%3ASafe&restrict_sr=on https://www.reddit.com/r/reinforcementlearning/search?q=flai... and https://www.reddit.com/r/reinforcementlearning/search?q=flair%3ADL+flair%3ABayes&restrict_sr=on&sort=relevance&t=all https://www.reddit.com/r/reinforcementlearning/search?q=flai...
- clickok 9y ago
  Yes. In particular, it's possible to learn the variance of the return using TD-methods with the same computational complexity as learning the expected value (the value function). See [0] for how to do it via the squared TD-error, or [1] for how to estimate it via the second moment of the return, and my own notes (soon to be published and expanded for my thesis) here [2]. It turns out that identifying states with high variance is a great way of locating model error-- most of the real-world environments are fairly deterministic, so states with high variance tend to be "aliased" combinations of different states with wildly different outcomes. You can use this to improve your agent via either allocating more representation power to those states to differentiate between very similar ones, or have your agent account for variance when choosing its policy. For example, Tesla could identify when variance spikes in its model and trigger an alert to the user that they may need to take over. Additionally, there's work by Bellemare [3] for estimating the distribution of the return, which allows for all sorts of statistical techniques for quantifying confidence, risk, or uncertainty. --- 0. https://arxiv.org/abs/1801.08287 https://arxiv.org/abs/1801.08287 1. https://arxiv.org/abs/1607.00446 https://arxiv.org/abs/1607.00446 2. http://rl.ai/posts/fun-with-the-td-error-part-ii.html http://rl.ai/posts/fun-with-the-td-error-part-ii.html 3. https://arxiv.org/abs/1707.06887 https://arxiv.org/abs/1707.06887
- hutec 9y ago
  Yes, there is an active branch in deep learning (Bayesian deep learning) trying to model uncertainty measurements into neural networks. Older ideas are http://mlg.eng.cam.ac.uk/yarin/blog_2248.html http://mlg.eng.cam.ac.uk/yarin/blog_2248.html or http://papers.nips.cc/paper/7219-simple-and-scalable-predictive-uncertainty-estimation-using-deep-ensembles.pdf http://papers.nips.cc/paper/7219-simple-and-scalable-predict... Basically Bayesian neural networks were able to model confidence but are not applicable in current real-world scenarios. Thus, lots of methods rely on approximating bayesian methods.
lisper 9y ago
> if a simple explanation exists, I'd love to hear it Basically it's a matter of having multiple redundant sensors and sounding the alarm if they don't all agree on what is going on, and also checking if what they think is going on is in the realm of reasonable possibility based on some a priori model (e.g. if suddenly all of your sensors tell you that you are 100 miles from where you were one second ago, that's probably a mistake even if all the sensors agree). That's a serious oversimplification, but that's the gist.
- deleted 9y ago
  [deleted]
- Someone1234 9y ago
  That's exactly how a lot of commercial aircraft systems work, they have three, if two agree and one disagree then that one is ignored. But it is more complicated than that when you're talking about algorithms and complex systems. If you had three of the same exact system, they'd likely all make the same mistake and thus you'd gain no safety improvement (except actual malfunctions, not logical limitations). I would like to see auto-braking taken out of autopilot completely, so if autopilot drives you into a wall at least the brakes slow you.
  - fma 9y ago
    https://en.wikipedia.org/wiki/Airborne_collision_avoidance_system https://en.wikipedia.org/wiki/Airborne_collision_avoidance_s... I'm not an expert...but it seems like for airplanes there are standards that manufacturers need to abide by for avoidance systems...which would mean testing and certification by an independent association before it's deployed. With manufacturers doing OTA updates...there's really no guarantee it's been tested...you'd have to trust your life to the QA process of each manufacturer. Not only the one of the car you're driving...but the car next to you!
  - ricardobeat 9y ago
    This has been mentioned a lot of times lately (the auto-braking). When you think about it, you realize there are a million moments during normal driving where you have an obstacle ahead. Other cars, lane dividers like these that you might _have_ to pass by closely, or at least will have your car pointed at it for a few seconds, making tight curves on a walled road, high curbs, buildings that are on the edge of the road, etc. It's not that simple, and I bet a lot of this is already taken into account by the software.

by dancek 9y ago

DSingularity 9y ago

I feel that’s the problem. Techniques like SVM can provide reasonable definitions of confidence. Reinforcement Learning.. DNN... the mechanisms behind autonomous cars... are they able to do this??

xiphias 9y ago
Neural networks are just non-linear functions that give probabilities to classes, so yes, it can be definitely done (and is done). As Tesla's computer vision system is not as good as Waymo's, it has to choose between comfort and safety more often.
mhermher 9y ago
not an expert, but aren't the output nodes thresholded to make a decision? How far you are from the threshold might be interpretable as confidence possibly?
- xapata 9y ago
  For a single perceptron, not for the network as a whole.
- CardenB 9y ago
  Yep that's exactly what you would do. However, in practice, this usually is just not a good indicator of confidence. NNs are notoriously overconfident.
gwern 9y ago
> Reinforcement Learning.. DNN... the mechanisms behind autonomous cars... are they able to do this?? Sorta. There are ways to extract kinds of uncertainty and confidence from NNs: for example, Gal's dropout trick where you train with dropout and then at runtime you use an 'ensemble' of multiple dropout-ed versions of your model, and the set of predictions gives a quasi-Bayesian posterior distribution for the predictions. NNs can be trained directly via HMC for small NNs, and there are arguments that constant-learning-rate SGD 'really' implements Bayesian inference and an ensemble of checkpoints yields an approximation of the posterior, etc. You can also train RL NNs which have an action of shortcutting computation and kicking the problem out to an oracle in exchange for a penalty, which trains them to specialize and 'know what they don't know' so they choose to call the oracle when they're insufficiently sure (this can be done for computational savings if the main NN is a small fast simple one and the oracle is a much bigger slower NN, or for safety if you imagine the oracle is a human or some fallback mechanism like halting). I have some cites on these sorts of things in https://www.gwern.net/Tool-AI https://www.gwern.net/Tool-AI and you could also look at the relevant tags https://www.reddit.com/r/reinforcementlearning/search?q=flair%3ASafe&restrict_sr=on https://www.reddit.com/r/reinforcementlearning/search?q=flai... and https://www.reddit.com/r/reinforcementlearning/search?q=flair%3ADL+flair%3ABayes&restrict_sr=on&sort=relevance&t=all https://www.reddit.com/r/reinforcementlearning/search?q=flai...
clickok 9y ago
Yes. In particular, it's possible to learn the variance of the return using TD-methods with the same computational complexity as learning the expected value (the value function). See [0] for how to do it via the squared TD-error, or [1] for how to estimate it via the second moment of the return, and my own notes (soon to be published and expanded for my thesis) here [2]. It turns out that identifying states with high variance is a great way of locating model error-- most of the real-world environments are fairly deterministic, so states with high variance tend to be "aliased" combinations of different states with wildly different outcomes. You can use this to improve your agent via either allocating more representation power to those states to differentiate between very similar ones, or have your agent account for variance when choosing its policy. For example, Tesla could identify when variance spikes in its model and trigger an alert to the user that they may need to take over. Additionally, there's work by Bellemare [3] for estimating the distribution of the return, which allows for all sorts of statistical techniques for quantifying confidence, risk, or uncertainty. --- 0. https://arxiv.org/abs/1801.08287 https://arxiv.org/abs/1801.08287 1. https://arxiv.org/abs/1607.00446 https://arxiv.org/abs/1607.00446 2. http://rl.ai/posts/fun-with-the-td-error-part-ii.html http://rl.ai/posts/fun-with-the-td-error-part-ii.html 3. https://arxiv.org/abs/1707.06887 https://arxiv.org/abs/1707.06887
hutec 9y ago
Yes, there is an active branch in deep learning (Bayesian deep learning) trying to model uncertainty measurements into neural networks. Older ideas are http://mlg.eng.cam.ac.uk/yarin/blog_2248.html http://mlg.eng.cam.ac.uk/yarin/blog_2248.html or http://papers.nips.cc/paper/7219-simple-and-scalable-predictive-uncertainty-estimation-using-deep-ensembles.pdf http://papers.nips.cc/paper/7219-simple-and-scalable-predict... Basically Bayesian neural networks were able to model confidence but are not applicable in current real-world scenarios. Thus, lots of methods rely on approximating bayesian methods.

lisper 9y ago

> if a simple explanation exists, I'd love to hear it Basically it's a matter of having multiple redundant sensors and sounding the alarm if they don't all agree on what is going on, and also checking if what they think is going on is in the realm of reasonable possibility based on some a priori model (e.g. if suddenly all of your sensors tell you that you are 100 miles from where you were one second ago, that's probably a mistake even if all the sensors agree). That's a serious oversimplification, but that's the gist.

deleted 9y ago
[deleted]
Someone1234 9y ago
That's exactly how a lot of commercial aircraft systems work, they have three, if two agree and one disagree then that one is ignored. But it is more complicated than that when you're talking about algorithms and complex systems. If you had three of the same exact system, they'd likely all make the same mistake and thus you'd gain no safety improvement (except actual malfunctions, not logical limitations). I would like to see auto-braking taken out of autopilot completely, so if autopilot drives you into a wall at least the brakes slow you.
- fma 9y ago
  https://en.wikipedia.org/wiki/Airborne_collision_avoidance_system https://en.wikipedia.org/wiki/Airborne_collision_avoidance_s... I'm not an expert...but it seems like for airplanes there are standards that manufacturers need to abide by for avoidance systems...which would mean testing and certification by an independent association before it's deployed. With manufacturers doing OTA updates...there's really no guarantee it's been tested...you'd have to trust your life to the QA process of each manufacturer. Not only the one of the car you're driving...but the car next to you!
- ricardobeat 9y ago
  This has been mentioned a lot of times lately (the auto-braking). When you think about it, you realize there are a million moments during normal driving where you have an obstacle ahead. Other cars, lane dividers like these that you might _have_ to pass by closely, or at least will have your car pointed at it for a few seconds, making tight curves on a walled road, high curbs, buildings that are on the edge of the road, etc. It's not that simple, and I bet a lot of this is already taken into account by the software.