8 ms·
An Introduction to Probabilistic Graphical Models (2003) [pdf]
- visarga 10y agoPGM seems to me harder than neural nets, but the trend in the last couple of years is to include probabilities in neural nets, so they're hot.
- adamnemecek 10y agoNot an expert but PGMs are mostly a superset of NN, so it's kinda understandable.
- ilzmastr 10y agoWhat is the connection other than "they both have graphs somewhere in them"? I sort of see what you mean since NNs transport information across a graph in a straightforward (non-loopy) way, and PGMs can propagate information in crazy (neverending loops) ways when doing belief propagation...
- adamnemecek 10y agoNot just graphs but also probability.
- visarga 10y agoFrom my limited understanding, pure NNs are not able to express confidence in predictions. By adding probability to the NN, we can have both predictions and confidence scores. In practice, noise is being injected, or connections dropped out randomly, then predictions averaged over multiple runs.
- ilzmastr 10y agoSure they can, many output probabilities: http://deeplearning.net/tutorial/mlp.html#mlp http://deeplearning.net/tutorial/mlp.html#mlp
- throwaway18974 10y agoThey are not.
- lowglow 10y agoI used "Probabilistic Graphical Models" By Koller/Friedman [0] https://www.amazon.com/Probabilistic-Graphical-Models-Principles-Computation/dp/0262013193/ https://www.amazon.com/Probabilistic-Graphical-Models-Princi...
- platz 10y agoPGM's are great, but my experience from Koller's course is that it is very hard to identify cases where they can be used. Part of the reason is that you need a-priori knowledge of the causal relationships (coarse grained I.e direction) between your variables. Presumably if you're doing ML you don't know those causal relationships to begin with. Particularly good fits are things like physics where laws are known.
- mamp 10y agoA heuristic guide...If you have more data than knowledge about the domain, and you want to do classification/prediction, then NNs are a good. PGMs are good if you have knowledge that is important to encode; you need modularity i.e. you don't want to embed priors in the model; you have strong causal relationships e.g. diagnostic models with 'explaining away'; you want to integrate value of information e.g. what test should I do next to resolve uncertainty. You can mix ML with knowledge in PGMs. The downside is the computational complexity of inference which is NP-hard for both exact and approximate inference, although you can identify the complexity of the model at design time and make the appropriate modelling trade-offs.
- wvbeaaoo 10y agoby complexity of inference do you mean the complexity of learning the structure of a model ? because inference on an existing PGM is linear in the number of edges with belief-propagation isn't it ?
- BucketSort 10y agoThere's an excellent course on PGM by Koller on Coursera. My friend took it and now he's a PGM evangelist. If you are wondering where PGM lies in the spectrum of machine learning, you should research the difference between generative and discriminate modeling. We have been driven to PGM to solve our ML problem that was hard to frame as A NN. Mainly because we had some priors we needed to encode to make the problem tractable. It reminds me a little of heuristics in search. The person I'm talking to: an early ML student.
- throwaway18974 10y agoThe hot use-case of the PGMs approach has often been in the discriminative setting (see M^3 and latent SVM) - which is good because discriminative classifiers work well with fewer data points (see Ng, Russel).
- tachim 10y agoThis is the best textbook on graphical models, also from Jordan but later (2008): https://people.eecs.berkeley.edu/~wainwrig/Papers/WaiJor08_FTML.pdf https://people.eecs.berkeley.edu/~wainwrig/Papers/WaiJor08_F.... It also covers some general theory of variational inference. Source: I worked on PGMs in grad school.
- diab0lic 10y agoA few comments have mentioned neural nets in this post. adamnemecek mentions in this thread that PGMs are a superset of neural networks, and and Thomas Wiecki has a few excellent blog posts on creating bayesian neural networks using pymc3.[0][1][2] If you're curious about how these two concepts can be brought together I highly recommend reading through these three posts. [0] http://twiecki.github.io/blog/2016/06/01/bayesian-deep-learning/ http://twiecki.github.io/blog/2016/06/01/bayesian-deep-learn... [1] http://twiecki.github.io/blog/2016/07/05/bayesian-deep-learning/ http://twiecki.github.io/blog/2016/07/05/bayesian-deep-learn... [2] http://twiecki.github.io/blog/2017/03/14/random-walk-deep-net/ http://twiecki.github.io/blog/2017/03/14/random-walk-deep-ne...
- ganfortran 10y agoDo they train with backpropagation efficiently?
- twiecki 10y agoNo, back-propagation would not give full Bayesian inference (although there are some tricks [0]). They instead use variational inference[1], which allows for fast inference of continuous PGMs. [0] http://mlg.eng.cam.ac.uk/yarin/blog_2248.html http://mlg.eng.cam.ac.uk/yarin/blog_2248.html [1] https://arxiv.org/abs/1603.00788 https://arxiv.org/abs/1603.00788
- cttet 10y agoMost variational inference are not full-Bayesian as well...
- iconvalleysil 10y agoMichael I. Jordan was a mentor to Andrew Ng. Probabilistic Graphical Models are the next frontier in AI after deep learning.
- KasianFranks 10y agoGood article: "Big-data boondoggles and brain-inspired chips are just two of the things we’re really getting wrong" - Michael I. Jordan ref: http://spectrum.ieee.org/robotics/artificial-intelligence/machinelearning-maestro-michael-jordan-on-the-delusions-of-big-data-and-other-huge-engineering-efforts http://spectrum.ieee.org/robotics/artificial-intelligence/ma...
- MrQuincle 10y agoFrom: http://spectrum.ieee.org/robotics/artificial-intelligence/machinelearning-maestro-michael-jordan-on-the-delusions-of-big-data-and-other-huge-engineering-efforts http://spectrum.ieee.org/robotics/artificial-intelligence/ma... Jordan: Well, humans are able to deal with cluttered scenes. They are able to deal with huge numbers of categories. They can deal with inferences about the scene: “What if I sit down on that?” “What if I put something on top of something?” These are far beyond the capability of today’s machines. Deep learning is good at certain kinds of image classification. “What object is in this scene?” I think Jordan refers here to Bayesian models that incorporate gravity, occlusion, and other such concepts. http://www.cv-foundation.org/openaccess/content_cvpr_2013/html/Jiang_Hallucinated_Humans_as_2013_CVPR_paper.html http://www.cv-foundation.org/openaccess/content_cvpr_2013/ht... e.g. postulates entire humans to improve scene understanding. What I get out of this: Deep learning has to be enriched with progress from other machine learning fields
- Chris2048 10y agoRelated?: https://www.coursera.org/specializations/probabilistic-graphical-models https://www.coursera.org/specializations/probabilistic-graph... http://openclassroom.stanford.edu/MainFolder/CoursePage.php?course=ProbabilisticGraphicalModels http://openclassroom.stanford.edu/MainFolder/CoursePage.php?...
- JustFinishedBSG 10y agoThis course also referred M.I Jordan book: http://imagine.enpc.fr/~obozinsg/teaching/mva_gm/fall2016/ http://imagine.enpc.fr/~obozinsg/teaching/mva_gm/fall2016/ One of the best course I have ever taken, F. Bach and G. Obozinski are incredible teachers.
- aflam 10y agoI couldn't agree more! This course features has slides, which many will prefer.
- deleted 10y ago[deleted]
- leecarraher 10y agoNice high level talk on Statistical Inference for Big Data by Jordan. Been one of my favorites since his LDA/PLSA papers in 2003 with Andrew NG. http://videolectures.net/colt2014_jordan_bigdata/?q=jordan http://videolectures.net/colt2014_jordan_bigdata/?q=jordan
- shurtler 10y agoNote there's an exploding literature that reads these models as causal models: http://ftp.cs.ucla.edu/pub/stat_ser/r350.pdf http://ftp.cs.ucla.edu/pub/stat_ser/r350.pdf