4 ms·

I feel like variational inference has never been described very well to an intro audience even having statistical basics. Is it a graduate level topic or is th

by platz 8y ago

I feel like variational inference has never been described very well to an intro audience even having statistical basics. Is it a graduate level topic or is there an intuitive course that teaches it to beginners?

klipt 8y ago
"variational inference" is perhaps an uninformative name. You can just think of it as - approximating the posterior using a nice parametric distribution, then - minimizing some error (typically KL Divergence) between your approximate posterior and the true posterior
- ced 8y ago
  Do you know _why_ KL divergence is minimized? I get that it gives a lower bound on the marginal likelihood, which is cool, but is that it? What are the alternatives?
  - ssivark 8y ago
    KL divergence is motivated nicely from an information/coding theory viewpoint. It's very closely related to Shannon-von Neumann entropy [1], and KL(P||Q) characterizes the efficiency of a code designed for a model distribution P, when applied to reality which is actually represented by Q. A lot of recent work focuses on the Wasserstein divergence [1] as an alternative. One advantage of Wasserstein over KL is that the Wasserstein metric provides better fit over the whole distribution instead of localizing on some specific regions, thereby preventing "mode collapse". This makes it a popular metric for training Generative Adversarial Networks (GANs). For recent work on applying Wasserstein distance to variational inference, see: https://arxiv.org/abs/1805.11284 https://arxiv.org/abs/1805.11284 [1]: https://physics.stackexchange.com/questions/64574/definition-of-the-entropy/64597#64597 https://physics.stackexchange.com/questions/64574/definition... [2]: https://en.wikipedia.org/wiki/Wasserstein_metric https://en.wikipedia.org/wiki/Wasserstein_metric
meatmanek 8y ago
I found this talk to be useful, despite the technical difficulties https://www.youtube.com/watch?v=Dv86zdWjJKQ https://www.youtube.com/watch?v=Dv86zdWjJKQ (I haven't watched https://www.youtube.com/watch?v=ogdv_6dbvVQ https://www.youtube.com/watch?v=ogdv_6dbvVQ but it seems like a longer version of the same talk)

by platz 8y ago

klipt 8y ago

"variational inference" is perhaps an uninformative name. You can just think of it as - approximating the posterior using a nice parametric distribution, then - minimizing some error (typically KL Divergence) between your approximate posterior and the true posterior

ced 8y ago
Do you know _why_ KL divergence is minimized? I get that it gives a lower bound on the marginal likelihood, which is cool, but is that it? What are the alternatives?
- ssivark 8y ago
  KL divergence is motivated nicely from an information/coding theory viewpoint. It's very closely related to Shannon-von Neumann entropy [1], and KL(P||Q) characterizes the efficiency of a code designed for a model distribution P, when applied to reality which is actually represented by Q. A lot of recent work focuses on the Wasserstein divergence [1] as an alternative. One advantage of Wasserstein over KL is that the Wasserstein metric provides better fit over the whole distribution instead of localizing on some specific regions, thereby preventing "mode collapse". This makes it a popular metric for training Generative Adversarial Networks (GANs). For recent work on applying Wasserstein distance to variational inference, see: https://arxiv.org/abs/1805.11284 https://arxiv.org/abs/1805.11284 [1]: https://physics.stackexchange.com/questions/64574/definition-of-the-entropy/64597#64597 https://physics.stackexchange.com/questions/64574/definition... [2]: https://en.wikipedia.org/wiki/Wasserstein_metric https://en.wikipedia.org/wiki/Wasserstein_metric

meatmanek 8y ago

I found this talk to be useful, despite the technical difficulties https://www.youtube.com/watch?v=Dv86zdWjJKQ https://www.youtube.com/watch?v=Dv86zdWjJKQ (I haven't watched https://www.youtube.com/watch?v=ogdv_6dbvVQ https://www.youtube.com/watch?v=ogdv_6dbvVQ but it seems like a longer version of the same talk)