3 ms·
What's interesting about most complaints of these systems is people talk about their poor performance or scalability? That is usually more a consequence of usin
by cf 11y ago
What's interesting about most complaints of these systems is people talk about their poor performance or scalability? That is usually more a consequence of using MCMC or other inference algorithm than the language itself.
MCMC is a very slow inference algorithm. Its primary advantage was that for well-known models it could be coded up much more simply than a fancier inference technique. When you consider variational methods and newer streaming methods based on things like Assumed Density Filtering you can get really great scalable performance. The point of probabilistic programming is write inference algorithms once for a large class of models and be done. So the advantage of using a fancier method is amplified.
This means paradoxically probabilistic programming should eventually be faster than existing methods rather than slower, since you can reuse these fancier inference methods for new models. This is a very active field so this progress is only starting to be appear in the existing systems.
- detaro 11y ago> The point of probabilistic programming is write inference algorithms once for a large class of models and be done. So the advantage of using a fancier method is amplified. So that should be something where a few standard libraries or toolsets should emerge that push out the "easy-to-implement" default choice? Are there any contenders yet? (As you might be able to tell, I don't really know anything about the field)
- cf 11y agoI know STAN and Figaro are going to push out inference methods like I mention, but my hope is eventually all of the ones mentioned in the article do this. I like thinking of this in terms of the standard library that needs to be built out. All the systems are making great progress in this regard.
- x0x0 11y agoEase of coding is not the primary advantage of mcmc. The (giant) disadvantage of variational inference is the user needs to derive equations and understand quite a bit of math, where gibbs or other samplers like those in bugs/openbugs/jags/stan can work with the factored distributions and require much less mathematical sophistication from users.
- mjw 11y agoI know they've traditionally been quite fiddly, but I'm pretty sure computers can be persuaded to help derive the maths for variational methods these days. Perhaps a more important difference is that MCMC, while slow, is exact in the limit. Variational methods won't converge to the true posterior no matter how long you run them. You'll converge to an approximate answer which depends on the particular variational form you choose to use.
- x0x0 11y agoAre you aware of any work (or researchers) working on that? I would be very interested. And I think your second sentence reinforces my point -- it makes variational methods either more fiddly, or require more understanding to use well. edit: here's one such (limited but nice) effort: http://ebonilla.github.io/papers/nguyen-bonilla-nips-2014.pdf http://ebonilla.github.io/papers/nguyen-bonilla-nips-2014.pd...
- davmre 11y agoMicrosoft's Infer.NET is essentially an automated system for variational (/expectation-propagation) inference. It implements primitives for common operations and distributions, and then uses the local structure of mean-field inference ("variational message passing": http://www.jmlr.org/papers/volume6/winn05a/winn05a.pdf http://www.jmlr.org/papers/volume6/winn05a/winn05a.pdf) to build up variational inference algorithms for arbitrary factor graphs. It's not infinitely flexible and doesn't solve all problems related to variational inference, but once you get used to it you can iterate quite quickly on model refinements, inference tweaks, etc. without any tedious derivations.
- x0x0 11y agothank you!
- cf 11y agoI think we are broadly in agreement. When I say coding up inference for a particular model, that includes deriving the equations and updates needed. This effort is nonzero for all inference methods, but is much lower for MCMC.