7 ms·
Stanford Lecture Notes on Probabilistic Graphical Models
- philipov 9y agoOpenCourseOnline, different Stanford professor: https://www.youtube.com/watch?v=WPSQfOkb1M8&list=PL50E6E80E8525B59C&index=1 https://www.youtube.com/watch?v=WPSQfOkb1M8&list=PL50E6E80E8... Carnegie-Mellon: https://www.youtube.com/watch?v=lcVJ_zsynMc&list=PLI3nIOD-p5aoXrOzTd1P6CcLavu9rNtC- https://www.youtube.com/watch?v=lcVJ_zsynMc&list=PLI3nIOD-p5...
- refrigerator 9y agoFor anyone interested, here are the materials for the Graphical Models course at Oxford: http://www.stats.ox.ac.uk/~evans/gms/index.htm http://www.stats.ox.ac.uk/~evans/gms/index.htm
- mrcactu5 9y agocourse like these make me wonder ... how much one can dress up basic probability. i think the answer is A LOT
- deleted 9y ago[deleted]
- Kinnard 9y agoWhat makes you say that?
- adamnemecek 9y agoIf you can derive all this just from probability my hat is off to you.
- deleted 9y ago[deleted]
- graycat 9y agoIn their "Probability review" at http://ermongroup.github.io/cs228-notes/preliminaries/probabilityreview/ http://ermongroup.github.io/cs228-notes/preliminaries/probab... I see two problems: (1) First Problem -- Sample Space Their definition of a sample space is "The set of all the outcomes of a random experiment. Here, each outcome ω can be thought of as a complete description of the state of the real world at the end of the experiment." The "complete description" part is not needed and even if included has meaning that is not clear. Instead, each possible experiment is one trial and one element in the set of all trials Ω. That's it: Ω is just a set of trials, and each trial is just an element of that set. There is nothing there about the outcomes of the trials. Next the text has "The sample space is Ω = {1, 2, 3, 4, 5, 6}." That won't work: Too soon will find that need an uncountably infinite sample space. Indeed an early exercise is that the set of all events cannot be countably infinite. Indeed, a big question was, can there be a sample space big enough to discuss random variables as desired? The answer is yes and is given in the famous Kolomogorov extension theorem. (2) Second Problem -- Notation An event A is an element of the set of all events F and a subset of the sample space Ω. Then a probability measure P or just a probability is a function P: F --> [0,1] that is, the closed interval [0,1]. So, we can write the probability of event A by P(A). Fine. Or, given events A and B, we can consider the event C = A U B and, thus, write P(C) = P(A U B). Fine. But the notes have P(1,2,3,4), and that is undefined in the notes and, really, in the rest of probability. Why? Because 1,2,3,4, is not an event. For the set of real numbers R, a real random variable X: Ω --> R (that is measurable with respect to the sigma algebra F and a specified sigma algebra in R, usually the Borel sets, the smallest sigma algebra containing the open sets, or the Lebesgue measurable sets). Then an event would be X in {1,2,3,4} subset of R or the set of all ω in Ω so that X(ω) in {1,2,3,4} or {ω| X(ω) in {1,2,3,4} } or the inverse image of {1,2,3,4} under X -- could write this all more clearly if had all of D. Knuth's TeX. in which case we could write P(X in {1,2,3,4}) When the elementary notation is bad, a bit tough to take the more advanced parts seriously. A polished, elegant treatment of these basics is early in Jacques Neveu, Mathematical Foundations of the Calculus of Probability, Holden-Day, San Francisco, 1965. Neveu was a student of M. Loeve at Berkeley, and can also see Loeve, Probability Theory, I and II, Springer-Verlag. A fellow student of Neveu at Berkeley under Loeve was L. Breiman, so can also see Breiman, Probability, SIAM. These notes are from Stanford. But there have long been people at Stanford, e.g., K. Chung, who have these basics in very clear, solid, and polished terms, e.g., Kai Lai Chung, A Course in Probability Theory, Second Edition, ISBN 0-12-174650-X, Academic Press, New York, 1974. K. L. Chung and R. J. Williams, Introduction to Stochastic Integration, Second Edition, ISBN 0-8176-3386-3, Birkhaüser, Boston, 1990. Kai Lai Chung, Lectures from Markov Processes to Brownian Motion, ISBN 0-387-90618-5, Springer-Verlag, New York, 1982.
- georgeek 9y agoAn amazing text on this topic is Martin Wainwright/Michael Jordan's Graphical Models, Exponential Families, and Variational Inference: https://people.eecs.berkeley.edu/~wainwrig/Papers/WaiJor08_FTML.pdf https://people.eecs.berkeley.edu/~wainwrig/Papers/WaiJor08_F...
- rsp1984 9y agoThanks for sharing. Somehow I find a lot of Computer Science texts published under the "Foundations and Trends" model of very high quality.
- signa11 9y agowow ! thank you :) this should be required reading/prereq, ideally, before taking the PGM course offered via coursera.
- georgeek 9y agoThis is more of a graduate-level text though. It's as close as you can currently get to Michael Jordan's forthcoming book on graphical models and ML.
- beambot 9y agoFor a practical application of using graphical models to "solve" a bayesian problem, I recommend Frank Dellaert's whitepaper, which cover Simultaneous Localization and Mapping (SLAM, a robotics algorithm) using similar techniques: https://research.cc.gatech.edu/borg/sites/edu.borg/files/downloads/gtsam.pdf https://research.cc.gatech.edu/borg/sites/edu.borg/files/dow...
- rsp1984 9y agoI'm not an expert in Probabilistic Graphical Models but I do know factor graphs well. I've also read quite a bit of the recent SLAM and VO stuff that came out of Dellaert's group. Here's the thing: Dellaert loves to write about SLAM from a viewpoint of probabilities and densities. I think he likes to see himself as a mathematician. However at the end of the day he's working with Gaussian noise assumptions, just like everyone else, and the solution is obtained by some form of Least Squares, again, like everyone else. I would really like it if he just cut all the probabilistic thicket and went straight to the core of how his methods improve the SOTA just in terms of how the friggin Least Squares problem is set up and solved (e.g. like here [1]). But I guess that would probably take away most of the "magic". [1] http://grail.cs.washington.edu/projects/mcba/pba.pdf http://grail.cs.washington.edu/projects/mcba/pba.pdf
- aub3bhat 9y agoYou cannot be more correct in your criticism, it feels as if there are a group of researchers who are just too enamored by beauty of "Bayesian stats and graphical models" that finding better solution of problem has become a secondary task for them. Plus a segment of academia loves this type of research.
- Xcelerate 9y ago> However at the end of the day he's working with Gaussian noise assumptions, just like everyone else, and the solution is obtained by some form of Least Squares, again, like everyone else. Well you need some kind of model to optimize, and finding the perfect one is a vastly more difficult problem that probably begins to incorporate some stuff from information theory (which starts to become rather a different field). I don't actually see the issue with researchers working on algorithms to solve the Gaussian noise versions of these problems — they can usually be extended to some other model relatively easily (for instance, rotation synchronization via Riemannian manifolds can incorporate a Pseudo-Huber loss function without much difficulty). Also, what is SOTA?
- likelynew 9y agoWe have a professor in our college, who uses this thing with great passion. I took two related courses under him. The problem with this thing is he uses his own mental image and notation in the lectures and examination. Even the internet seems to be highly affected with this problem. I think there are many concepts that are not hard but it takes time to get a feeling. Like see "monads are not burritos" essay. It describes the problem of using analogies in explaining monads. This seems like a great page in the sense it uses minimal confusing analogies as is common in most resources in bayesian rule.
- dirtyaura 9y agoFor a novice, it's hard to assess how important PGMs are currently. Investing my time to learn Deep Learning (CNNs, RNNs) vs. Random Forest vs. PGMs vs. Reinforcement Learning well enough to be able to apply the chosen approach, it seems that PGMs are not high in the list, is that correct? Are there Kaggle competitions, in which PGMs have been the best approach? What are the real-world problem areas that PGM's currently excel at compared to other methods?
- eachro 9y agoRandom Forests is not really at all in the same class of depth(no pun intended) as the other models you've mentioned; if you understand decision trees, random forests are just a way of combining independently trained decision trees that is less prone to overfitting. I'd probably rank these areas in order of importance as follows: deep learning, pgms, reinforcement learning. Deep learning as a framework is pretty general. PGMs, as I have seen them, don't really have any one killer domain area - maybe robotics and areas where you want to explicitly model causality? Applications for reinforcement learning seem the more niche, but maybe that's because they haven't been adequately explored to the extent that DL/CNNs/RNNs/PGMs have been.
- digitalzombie 9y agoTree based algorithm have it's place. I dunno what your definition of depth is but it all depends on your data. I'm doing a thesis on tree based algorithm and it works great for medical data. Granted I have little exposer to NN, you can't do that with NN when clinical trials data is small as hell. It's all base on the type data and your resource and criteria. NN is really hype and people tend to over look other algorithms but NN is not a silver bullet. I don't get how you rank those importance either, it could be bias.
- eachro 9y agoOh there's no question that tree based methods are effective - random forests/gradient boosted trees routinely win kaggle competitions. But I was more referring to how random forests is learnable in a day, whereas deep learning would probably take at least a few weeks to learn properly.