3 ms·

Ease of coding is not the primary advantage of mcmc. The (giant) disadvantage of variational inference is the user needs to derive equations and understand qui

by x0x0 12y ago

Ease of coding is not the primary advantage of mcmc. The (giant) disadvantage of variational inference is the user needs to derive equations and understand quite a bit of math, where gibbs or other samplers like those in bugs/openbugs/jags/stan can work with the factored distributions and require much less mathematical sophistication from users.

mjw 12y ago
I know they've traditionally been quite fiddly, but I'm pretty sure computers can be persuaded to help derive the maths for variational methods these days. Perhaps a more important difference is that MCMC, while slow, is exact in the limit. Variational methods won't converge to the true posterior no matter how long you run them. You'll converge to an approximate answer which depends on the particular variational form you choose to use.
- x0x0 12y ago
  Are you aware of any work (or researchers) working on that? I would be very interested. And I think your second sentence reinforces my point -- it makes variational methods either more fiddly, or require more understanding to use well. edit: here's one such (limited but nice) effort: http://ebonilla.github.io/papers/nguyen-bonilla-nips-2014.pdf http://ebonilla.github.io/papers/nguyen-bonilla-nips-2014.pd...
  - davmre 12y ago
    Microsoft's Infer.NET is essentially an automated system for variational (/expectation-propagation) inference. It implements primitives for common operations and distributions, and then uses the local structure of mean-field inference ("variational message passing": http://www.jmlr.org/papers/volume6/winn05a/winn05a.pdf http://www.jmlr.org/papers/volume6/winn05a/winn05a.pdf) to build up variational inference algorithms for arbitrary factor graphs. It's not infinitely flexible and doesn't solve all problems related to variational inference, but once you get used to it you can iterate quite quickly on model refinements, inference tweaks, etc. without any tedious derivations.
    - x0x0 12y ago
      thank you!
cf 12y ago
I think we are broadly in agreement. When I say coding up inference for a particular model, that includes deriving the equations and updates needed. This effort is nonzero for all inference methods, but is much lower for MCMC.
- x0x0 12y ago
  You're totally right; I should have read it that way.

by x0x0 12y ago

mjw 12y ago

I know they've traditionally been quite fiddly, but I'm pretty sure computers can be persuaded to help derive the maths for variational methods these days. Perhaps a more important difference is that MCMC, while slow, is exact in the limit. Variational methods won't converge to the true posterior no matter how long you run them. You'll converge to an approximate answer which depends on the particular variational form you choose to use.

x0x0 12y ago
Are you aware of any work (or researchers) working on that? I would be very interested. And I think your second sentence reinforces my point -- it makes variational methods either more fiddly, or require more understanding to use well. edit: here's one such (limited but nice) effort: http://ebonilla.github.io/papers/nguyen-bonilla-nips-2014.pdf http://ebonilla.github.io/papers/nguyen-bonilla-nips-2014.pd...
- davmre 12y ago
  Microsoft's Infer.NET is essentially an automated system for variational (/expectation-propagation) inference. It implements primitives for common operations and distributions, and then uses the local structure of mean-field inference ("variational message passing": http://www.jmlr.org/papers/volume6/winn05a/winn05a.pdf http://www.jmlr.org/papers/volume6/winn05a/winn05a.pdf) to build up variational inference algorithms for arbitrary factor graphs. It's not infinitely flexible and doesn't solve all problems related to variational inference, but once you get used to it you can iterate quite quickly on model refinements, inference tweaks, etc. without any tedious derivations.
  - x0x0 12y ago
    thank you!

cf 12y ago

I think we are broadly in agreement. When I say coding up inference for a particular model, that includes deriving the equations and updates needed. This effort is nonzero for all inference methods, but is much lower for MCMC.

x0x0 12y ago
You're totally right; I should have read it that way.