9 ms·

Thanks for your clarification! I'm not a statistician, I just happen to be surrounded by them :) My impression is that among the Bayesians I know there is a ge

by ot 14y ago

Thanks for your clarification! I'm not a statistician, I just happen to be surrounded by them :) My impression is that among the Bayesians I know there is a general negative bias towards MAP, and Variational methods are vastly preferred. However I agree with you that, all being approximations, none of them is intrinsically better than the others. In particular I don't understand all the hype around Variational Bayes, to me it seems like a "fat MAP", a MAP estimate with a Gaussian around it.

ced 14y ago
Right - MAP may (debatably) be a lousy Bayesian approximation, but it's still Bayesian :) David MacKay's wonderful book made the observation that MAP is a variational method that uses a delta function. "From this perspective, any approximating distribution Q(x; θ) [like the Gaussian], no matter how crummy it is, has to be an improvement on the spike produced by the standard method! [MAP]"
- ot 14y ago
  Awesome observation! I skimmed through his book, but didn't notice this quote.
Dn_Ab 14y ago
I've only recently come across the technique myself. I think the hype is because it is new (well new in old new thing kind of way). What I find interesting is the duality like relationship between MCMC and variational methods. Variational methods are an optimization alogrithm. I don't understand variational methods enough to say anything insightful but given the work showing the duality between optimization and probability* I find this new development of hype to be highly interesting. Drawing on the concept of duality, I think Variational methods will come to be seen as holding no more power than probabilistic techniques. But a Duality is still great cause you can plumb old techniques to get new results. *http://www.cmap.polytechnique.fr/~akian/publis/akian_quadrat_viot_98.pdf http://www.cmap.polytechnique.fr/~akian/publis/akian_quadrat...

by ot 14y ago

ced 14y ago

Right - MAP may (debatably) be a lousy Bayesian approximation, but it's still Bayesian :) David MacKay's wonderful book made the observation that MAP is a variational method that uses a delta function. "From this perspective, any approximating distribution Q(x; θ) [like the Gaussian], no matter how crummy it is, has to be an improvement on the spike produced by the standard method! [MAP]"

ot 14y ago
Awesome observation! I skimmed through his book, but didn't notice this quote.

Dn_Ab 14y ago

I've only recently come across the technique myself. I think the hype is because it is new (well new in old new thing kind of way). What I find interesting is the duality like relationship between MCMC and variational methods. Variational methods are an optimization alogrithm. I don't understand variational methods enough to say anything insightful but given the work showing the duality between optimization and probability* I find this new development of hype to be highly interesting. Drawing on the concept of duality, I think Variational methods will come to be seen as holding no more power than probabilistic techniques. But a Duality is still great cause you can plumb old techniques to get new results. *http://www.cmap.polytechnique.fr/~akian/publis/akian_quadrat_viot_98.pdf http://www.cmap.polytechnique.fr/~akian/publis/akian_quadrat...