3 ms·
Whereas a frequentist model looks at an absolute basis for chances, something like the population of females is 52%, so that means that if I select someone
by ot 14y ago
Whereas a frequentist model looks at an absolute basis for
chances, something like the population of females is 52%, so that
means that if I select someone at random from my office, I have a
52% chance of picking a female. The chances are purely based on
the total probability. The Bayesian approach is to rely on past
knowledge and then adjust accordingly. If I know that 75% of my
office is male, and I grab a person, then I know that I have a
25% chance of picking a female.
This is a terrible example, it makes it look like frequentist statistics don't know about conditional probabilities.
My understanding is that frequentist statistics are all about point estimates (such as maximum likelihood) or sometimes confidence intervals. Say you want to run an elections poll, sample a bunch of people and ask who they have voted for.
Frequentists will average the data and say "Party A is at 51%" (point estimate) or "Party A is between 49% and 52%" (confidence interval). Asymptotically and under certain assumptions this value will converge to the "real" value and often you can also estimate the speed of convergence (with variance bounds).
Bayesians will instead start with a "prior" which is a probability distribution p(x) on "Party A is at x%". You can start from an uniform, non-informative prior, or if you have some information you can factor it into the prior. Then you take your polls data D and compute the conditional probability p(x | D), called "posterior", which is the probability that "Party A is at x% given the data". So you don't get a number or an uniform interval, you get a probability for each possible electoral outcome. Again, if you have infinite data this will converge to a distribution where all the mass is on a single point, which usually is the same given by frequentist statistics.
The problem with Bayesian statistics is that you have to handle probability distributions instead of single numbers (as with point estimates), so the inference gets extremely harder. It has been made somewhat possible only recently thanks to both algorithmic and hardware advances. On the other hand, the main advantage is that you get to know how uncertain your estimate is, which can make a huge difference when you have little data.
Last note: another thing you can do when you have a prior is factor it into your estimator and take the maximum likelihood of the posterior as a point estimate. This is called Maximum a Posteriori (MAP) and called by some people "Bayesian", but I don't think Bayesians agree with that.
- ced 14y agoLast note: another thing you can do when you have a prior is factor it into your estimator and take the maximum likelihood of the posterior as a point estimate. This is called Maximum a Posteriori (MAP) and called by some people "Bayesian", but I don't think Bayesians agree with that. The general perspective is that, of course, you'd like to get the fully marginalized, exact posterior probability distribution. However, that's not computationally viable for most problems, so you have to resort to approximations, like MCMC, Variational methods and MAP. I would say that they're all definitely "Bayesian" methods, as long as you're aware of what you're doing, and that you check the quality of your approximation.
- ot 14y agoThanks for your clarification! I'm not a statistician, I just happen to be surrounded by them :) My impression is that among the Bayesians I know there is a general negative bias towards MAP, and Variational methods are vastly preferred. However I agree with you that, all being approximations, none of them is intrinsically better than the others. In particular I don't understand all the hype around Variational Bayes, to me it seems like a "fat MAP", a MAP estimate with a Gaussian around it.
- ced 14y agoRight - MAP may (debatably) be a lousy Bayesian approximation, but it's still Bayesian :) David MacKay's wonderful book made the observation that MAP is a variational method that uses a delta function. "From this perspective, any approximating distribution Q(x; θ) [like the Gaussian], no matter how crummy it is, has to be an improvement on the spike produced by the standard method! [MAP]"
- ot 14y agoAwesome observation! I skimmed through his book, but didn't notice this quote.
- Dn_Ab 14y agoI've only recently come across the technique myself. I think the hype is because it is new (well new in old new thing kind of way). What I find interesting is the duality like relationship between MCMC and variational methods. Variational methods are an optimization alogrithm. I don't understand variational methods enough to say anything insightful but given the work showing the duality between optimization and probability* I find this new development of hype to be highly interesting. Drawing on the concept of duality, I think Variational methods will come to be seen as holding no more power than probabilistic techniques. But a Duality is still great cause you can plumb old techniques to get new results. *http://www.cmap.polytechnique.fr/~akian/publis/akian_quadrat_viot_98.pdf http://www.cmap.polytechnique.fr/~akian/publis/akian_quadrat...