9 ms·
The Case for Bayesian Deep Learning
- sgstfevsh 7y agoFor folks that want to start making sense of what's being said I recommend going through d2l.ai, fast.ai, and deeplearningbook.org. Most of contemporary AI is really just a combination of probability theory and some upper division math classes in an executable form. None of it is magic and the more people that know the vocabulary the less likely people are to buy into the hype. If you want a high level overview of all this then Melanie Mitchell has a good book as well: https://melaniemitchell.me/aibook/ https://melaniemitchell.me/aibook/. She does a really good job of putting everything into the right context and dispelling the marketing hype about the coming singularity and human obsolescence. In one of the chapters she covers deep reinforcement learning and it's one of the best high level explanations I've come across yet.
- fizixer 7y ago> ... just a combination of probability theory and some upper division math classes ... wow thanks for opening my eyes. I guess I should just get together with my undergrad friends, implement the next DeepMind competitor, and collect our $400m from Google.
- eli_gottlieb 7y agoWell... yeah, kinda. Depth of domain knowledge and good engineering practice remains more important than individual genius in getting these things to actually work.
- sgt101 7y agoAgree - also business and people management. The funds must flow, the people must stay, and work. Together. Vision (the right vision) is a huge differentiator and ingredient x. I think you (I) can pull together a team with knowledge, practice, management and people structures, and make it work. The folks who do epoch making stuff have the amazing vision thing as well.
- jbay808 7y ago> Most of contemporary AI is really just a combination of probability theory and some upper division math classes in an executable form. None of it is magic and the more people that know the vocabulary the less likely people are to buy into the hype For those of us who think probability theory captures a large fraction of what intelligence does, this reads a lot like "don't believe the hype behind nuclear weapons! It's really just physics and some upper division chemistry in an explosive form. None of it is magic..." Maybe this argument will convince people who think that the human brain is magic? But for those of us who think that the human brain is not magic, but proof that general intelligence is actually computable and only needs 20 watts... If probability theory and upper division math in executable form is all it takes to be dangerous, that wouldn't be surprising. It only takes physics and upper division chemistry in explosive form to level a city, after all. Not magic.
- whataretheodds 7y agoWhat do you suppose is dangerous about linear algebra and basic multivariable calculus? I personally don't think there is any danger in more people knowing more linear algebra and calculus.
- IAmEveryone 7y agoThe nuclear bombs were an analogy, and the point it is trying to prove is not "danger", but "powerful".
- jbay808 7y agoI use the term in the sense of the common phrase "know enough to be dangerous".
- sgt101 7y agoI'm setting up a reading group at work. Would this book be a good one for that? Any other recommendations?
- fhfhhf1233 7y agoThe Melanie Mitchell's book is mostly a high level overview. It's a really good introduction to the field but you will not become a good practitioner by just reading the book. If you want the mathematical details then d2l.ai so far seems pretty good but they use MXNet for the code. fast.ai uses PyTorch and in the later lessons introduces Swift4Tensorflow which bakes differentiable programming into the compiler. PyTorch is more popular so more jobs are likely to have code bases that use PyTorch. So I'd say it depends on your use cases. If you want high level understanding then go with Melanie Mitchell's book. If you want more technical details then d2l.ai or fast.ai are good choices. There's also "Deep Reinforcement Learning Hands-On" by Maxim Lapan. That one is also pretty good but I'm not far enough along to give a review one way or another.
- sgt101 7y agoThank you - good tips.
- lars 7y agoWhile the sources you mention are all good, I don't think any of them actually have much to say about what the link is about, which is bayesian deep learning.
- fhfhhf1233 7y agoThat's correct but the people that are interested need a place to start and those sources are as good a place as any. If you have good learning material for Bayesian approaches then you should post them because whoever is reading the comments might take note.
- clircle 7y agoI'm a self-described Bayesian* at my day job, but the author needs to do better to convince me that the Bayesian approach is worth it in the deep learning space. As far as I can tell, deep learning folks don't give two shits about uncertainty intervals, much less marginalization. All that matters is minimizing that test error as fast as possible. So what if you get a posterior for each parameter... Who cares about the parameters in a neural network as long as the predictions seem well calibrated? The most convincing rationale for adopting a Bayesian perspective is contained the collected works of Jim Berger, which I see is cited by the author... but not used in the manuscript. * Of course, a Bayesian is just a statistician that uses Bayesian techniques even when it's not appropriate -- Andrew Gelman
- MiroF 7y agoI agree - far too often I just see "safety critical applications" thrown around and that is definitely not enough of a justification in my mind. There are alternative ways of getting uncertainty estimates that are far less work.
- ssivark 7y ago> As far as I can tell, deep learning folks don't give two shits about uncertainty intervals, much less marginalization. All that matters is minimizing that test error as fast as possible. So what if you get a posterior for each parameter... Who cares about the parameters in a neural network as long as the predictions seem well calibrated? Well, the one thing that they ought to actually give a shit about is generalization. All else (Bayesian or not) is only in service of that. Deep learning systems require too much data (and often not raw data), crazy amounts of compute, and still fail under comically irrelevant tweaks of the problem (eg: adversarial noise). Yes, the PAC learning model starts off by assuming that the test set had the same “data distribution” as the train set. It’s obviously not exactly the same (otherwise learning is kinda irrelevant), but deep learning seems incredibly fragile in its ability to generalize. It seems to require crazy amounts of domain randomization during training to avoid overfitting. Eg: The recent OpenAI Rubik’s cube demonstration (while an impressive achievement) was an indication of the fundamental silliness of deep RL (at least as of now). To solve the problem, they had to use classical Rubik’s cube solving algorithms, and completely model the physics of the situation (including friction on the surfaces!) and use that model to generate training data while varying every parameter for domain randomization. All that structured prior knowledge is absolutely unfeasible for even a slightly more complicated situation. And after all that, the robot had something like a 20% success rate IIRC. Impressive work on the team’s part, no doubt... but the conceptual framework does not inspire confidence.
- riku_iki 7y agoDid any model within bayesian DL achieve SOTA in any more or less popular benchmark?
- benrbray 7y agoI interned for a team working in this area, and my supervisor was quite happy with their results using Bayesian Deep Learning on ImageNet: https://arxiv.org/abs/1906.02506 https://arxiv.org/abs/1906.02506
- sixdimensional 7y agoDoes anybody on this thread know about Bayesian subjective probability? I have been fascinated by it for years, and have recently come to wonder if it represents something interesting in the way it factors expert knowledge with an easily explainable mathematical/probability based approach. I love the idea of man and machine working together and wonder.
- clircle 7y agoDo you have a specific question?
- AlexCoventry 7y ago> The BMA [Bayesian Model Average, or total probability] represents epistemic uncertainty — that is, uncertainty over which setting of weights (hypothesis) is correct, given limited data It's easy to come up with model families where a given data set has a high total probability because it has high probability in every model in the family, so the total probability on its own cannot function as a general epistemological measurement. Unless you're already modeling a neural net it's extremely unlikely that the model family represented by your Bayesian Deep Learning system includes anything representing the actual data-generating process. It's not just me saying this; Gelman & Shalizi point this out in their "Philosophy and the Practice of Bayesian Statistics": > ...it is hard to claim that the prior distributions used in applied work represent statisticians’ states of knowledge and belief before examining their data, if only because most statisticians do not believe their models are true, so their prior degree of belief in all of ϴ is not 1 but 0. The prior distribution is more like a regularization device, akin to the penalization terms added to the sum of squared errors when doing ridge regression and the lasso (Hastie, Tibshirani, & Friedman, 2009) or spline smoothing (Wahba, 1990) (Although they're only talking about the prior here, it applies equally well to the total probability, which is just the mean of the data given the prior.) It makes some sense to talk about Bayesian methods quantifying epistemological information when you have some good reason to believe that some portion of your model family accurately captures the data-generating process, or at least the parts of that process you care about and are relevant for the predictions you want to make. But that's almost never the case for non-parametric methods.
- deleted 7y ago[deleted]
- mjburgess 7y agoIt seems like what you're saying is: Bayesian methods may be used as models of causal processes, or as mere mechanisms to generate predictions. And in the former case, models will be parameterized by meaningful causal variables & their effect-strength; in the latter case, parameters have no explanatory role. And finally that: only in the explanatory case can a bayesian model be interpreted epistemically. I think I agree with this -- the relevant epistemic interpretation of a model is how well it fits to the world -- NOT -- to data! Data is the means by which models are selected. So if a model is not explanatory (ie., about the world) there is no sense in which it "fits"; and thus no epistemic interpretation.
- benrbray 7y agoRelated: My former supervisor (Dr. Emtiyaz Khan @ RIKEN AIP) held a NeurIPS 2019 Tutorial on "Deep Learning with Bayesian Principles". He gives a nice high-level overview of this area of research.
- 1024core 7y agoIn today's ML world, theory follows practice. You have to demonstrate SOTA results in a problem area before people will pay attention to your theory. So, if you're proposing some theory, first implement it and demonstrate it. Otherwise, it's just vaporware.
- etaioinshrdlu 7y agoSo to me, deep learning means learning very complex functions from data. Sometimes those functions work as classifiers, but other times they are just peculiar functions between two domains (imagine style transfer models or GANs). Is Bayesian Deep Learning worth looking into if I don't have much interest in statistical applications but much more just using deep learning as general purpose functions? For that matter, what about causal inference?
- nestorD 7y ago> Is Bayesian Deep Learning worth looking into if I don't have much interest in statistical applications but much more just using deep learning as general purpose functions? If you manage to build a model with a high accuracy and a sensible uncertainty on the output, you can use that information to do a lot of great things such as : - obviously apply the method to domains that require uncertainty for legal or technical reasons (simulation) - adding sample to improve you knowledge around uncertain input (active learning) - use an optimized betting strategy that takes risks into account (bayesian optimization) Gaussian process are a prime example of that but I am not aware of a deep learning approach that realises this (yet).
- nestorD 7y agoI stand corrected, after some research, I found a paper that offers a promising way to get good uncertainty information from a deep learning model : http://www.cs.ox.ac.uk/people/yarin.gal/website/blog_3d801aa532c1ce.html http://www.cs.ox.ac.uk/people/yarin.gal/website/blog_3d801aa...