4 ms·

> As far as I can tell, deep learning folks don't give two shits about uncertainty intervals, much less marginalization. Maybe folks making snapchat filters do

> As far as I can tell, deep learning folks don't give two shits about uncertainty intervals, much less marginalization. Maybe folks making snapchat filters don't care, but this is absolutely vital if you're doing something with a low margin of error (self-driving cars, financial work, etc...). If your neural net can tell you if it thinks it could be making an error, that information is absolutely vital for keeping your system from messing up big time. The problem is that none of the Bayesian methods for machine learning work well and quickly (please do correct me if I'm wrong--I will become my boss's favorite person). If they did, many many practitioners would be very excited to use them.

jacquesm 7y ago
Bayesian methods work reasonably well as long as the parameter space is relatively limited and you have enough data to extract some advantage but not enough to power a deep learning model. You also usually do not rely on Bayesian methods exclusively to solve some problem but use it in conjunction with other methods (sanity checks) to ensure you aren't outputting nonsense.
- MiroF 7y ago
  Ah yes, the well-known limited parameter space of deep neural networks
eli_gottlieb 7y ago
>The problem is that none of the Bayesian methods for machine learning work well and quickly (please do correct me if I'm wrong--I will become my boss's favorite person). If they did, many many practitioners would be very excited to use them. There are loads of methods based on variational autoencoders that seem to work fairly well for various problems.
nestorD 7y ago
After a bit of search I found something that appears to work both well and quickly: http://www.cs.ox.ac.uk/people/yarin.gal/website/blog_3d801aa532c1ce.html http://www.cs.ox.ac.uk/people/yarin.gal/website/blog_3d801aa... The basic idea is brilliant: take a deep neural network with dropout, keep the dropout at evaluation time, run it several times and deduce the uncertainty from the observed variance (with a bit of math). This lets you add an uncertainty estimation to the output of any deep neural network that uses dropout.
- lostdog 7y ago
  Yes, this is the best I've seen too. Unfortunately, "quickly" is a matter of perspective. Usually you train a neural net that's as large as possible with a runtime that's within your latency and GPU budget. Running the net even twice will blow up your budget. Also, I haven't seen a great evaluation on how well this method works in practice. For example, how well does predicted uncertainty capture mistakes on a real-world dataset?

jacquesm 7y ago

Bayesian methods work reasonably well as long as the parameter space is relatively limited and you have enough data to extract some advantage but not enough to power a deep learning model. You also usually do not rely on Bayesian methods exclusively to solve some problem but use it in conjunction with other methods (sanity checks) to ensure you aren't outputting nonsense.

MiroF 7y ago
Ah yes, the well-known limited parameter space of deep neural networks

eli_gottlieb 7y ago

>The problem is that none of the Bayesian methods for machine learning work well and quickly (please do correct me if I'm wrong--I will become my boss's favorite person). If they did, many many practitioners would be very excited to use them. There are loads of methods based on variational autoencoders that seem to work fairly well for various problems.

nestorD 7y ago

After a bit of search I found something that appears to work both well and quickly: http://www.cs.ox.ac.uk/people/yarin.gal/website/blog_3d801aa532c1ce.html http://www.cs.ox.ac.uk/people/yarin.gal/website/blog_3d801aa... The basic idea is brilliant: take a deep neural network with dropout, keep the dropout at evaluation time, run it several times and deduce the uncertainty from the observed variance (with a bit of math). This lets you add an uncertainty estimation to the output of any deep neural network that uses dropout.

lostdog 7y ago
Yes, this is the best I've seen too. Unfortunately, "quickly" is a matter of perspective. Usually you train a neural net that's as large as possible with a runtime that's within your latency and GPU budget. Running the net even twice will blow up your budget. Also, I haven't seen a great evaluation on how well this method works in practice. For example, how well does predicted uncertainty capture mistakes on a real-world dataset?