4 ms·
"variational inference" is perhaps an uninformative name. You can just think of it as - approximating the posterior using a nice parametric distribution, then
by klipt 8y ago
"variational inference" is perhaps an uninformative name. You can just think of it as
- approximating the posterior using a nice parametric distribution, then
- minimizing some error (typically KL Divergence) between your approximate posterior and the true posterior
- ced 8y agoDo you know _why_ KL divergence is minimized? I get that it gives a lower bound on the marginal likelihood, which is cool, but is that it? What are the alternatives?
- ssivark 8y agoKL divergence is motivated nicely from an information/coding theory viewpoint. It's very closely related to Shannon-von Neumann entropy [1], and KL(P||Q) characterizes the efficiency of a code designed for a model distribution P, when applied to reality which is actually represented by Q. A lot of recent work focuses on the Wasserstein divergence [1] as an alternative. One advantage of Wasserstein over KL is that the Wasserstein metric provides better fit over the whole distribution instead of localizing on some specific regions, thereby preventing "mode collapse". This makes it a popular metric for training Generative Adversarial Networks (GANs). For recent work on applying Wasserstein distance to variational inference, see: https://arxiv.org/abs/1805.11284 https://arxiv.org/abs/1805.11284 [1]: https://physics.stackexchange.com/questions/64574/definition-of-the-entropy/64597#64597 https://physics.stackexchange.com/questions/64574/definition... [2]: https://en.wikipedia.org/wiki/Wasserstein_metric https://en.wikipedia.org/wiki/Wasserstein_metric