3 ms·

I've tried to use probabilistic programming for building a model of real data, and it seems that there is a long way to go before it's practical and fast. On o

by mizzao 12y ago

I've tried to use probabilistic programming for building a model of real data, and it seems that there is a long way to go before it's practical and fast. On one hand, there are the Monte Carlo-based methods that will support modeling almost any distributions, but are slow to use for large amounts of data. On the other hand, there are interesting cases like Infer.NET that use a completely different technique (approximate, deterministic inference) but are brittle for many real-world use cases. Then, there is the general issue that one has to be familiar with probabilistic models and the inner workings of the inference algorithms to have any hope of debugging the inevitable errors and convergence issues that arise. That seems to realistically require a machine learning or statistics PhD and the population of those is very small.

gh02t 12y ago
The performance of most vanilla Metropolis-Hastings MCMC methods is pretty awful, but there are some good improvements. I use DREAM (a non-Markovian and parallelizable extension of M-H) with realistic problems and its convergence rate is much much much better, to the point that I can use it for what I'm working on while regular M-H methods are just too slow.
stdbrouw 12y ago
Well, MCMC is pretty close to brute force so it's never going to be very fast. But the diagnostic tools are really quite good these days – tools like Stan and PyMC make it easy to see diagnostic plots that allow you to check for convergence and such. Additionally, more and more fine-tuning is happening automatically, like setting the right jump sizes and a plausible initial value for the random walk. Note that traditional statistics, like regression, requires similar diagnostic steps: qqplots to check for the normality of your data, the Levene test for heteroskedasticity and so on. They often get ignored, but they're generally not hard to teach or learn. We're really not that far from having this stuff be usable by the broader public. Maybe not "build any model you like and it will just work out of the box" but definitely as a replacement for much frequentist statistics. See "Doing Bayesian Data Analysis" by Kruschke, for example.
- x0x0 12y ago
  I'm more pessimistic. My experience has been that inferring often benefits from reparameterization or analytic marginalizing, plus a bag of tricks that many people don't know. It's unfortunately still a long way from being a technology that you can pretty blindly use like ols or logistic regression. Particularly if you only care about prediction, not parameter inference, lr with a penalty search is pretty straightforward. May I ask what / if you use bayesian stats for professionally in your news work? Also, the link to your github from your site goes to stdbrouw instead of debrouwere
  - stdbrouw 12y ago
    So do you feel the pitfalls of probabilistic programming / Bayesian data analysis are potentially more severe than those of traditional methods? I don't get to use that much Bayesian analysis in my work, but I've been toying around with a hierarchical model of content popularity recently: are some topics / authors / genres more popular than others? Frankly finding it hard to separate the signal ("this is what people actually like to read") from the noise ("this is what just happened to be trending and will not be reproducible").

by mizzao 12y ago

gh02t 12y ago

The performance of most vanilla Metropolis-Hastings MCMC methods is pretty awful, but there are some good improvements. I use DREAM (a non-Markovian and parallelizable extension of M-H) with realistic problems and its convergence rate is much much much better, to the point that I can use it for what I'm working on while regular M-H methods are just too slow.

stdbrouw 12y ago

Well, MCMC is pretty close to brute force so it's never going to be very fast. But the diagnostic tools are really quite good these days – tools like Stan and PyMC make it easy to see diagnostic plots that allow you to check for convergence and such. Additionally, more and more fine-tuning is happening automatically, like setting the right jump sizes and a plausible initial value for the random walk. Note that traditional statistics, like regression, requires similar diagnostic steps: qqplots to check for the normality of your data, the Levene test for heteroskedasticity and so on. They often get ignored, but they're generally not hard to teach or learn. We're really not that far from having this stuff be usable by the broader public. Maybe not "build any model you like and it will just work out of the box" but definitely as a replacement for much frequentist statistics. See "Doing Bayesian Data Analysis" by Kruschke, for example.

x0x0 12y ago
I'm more pessimistic. My experience has been that inferring often benefits from reparameterization or analytic marginalizing, plus a bag of tricks that many people don't know. It's unfortunately still a long way from being a technology that you can pretty blindly use like ols or logistic regression. Particularly if you only care about prediction, not parameter inference, lr with a penalty search is pretty straightforward. May I ask what / if you use bayesian stats for professionally in your news work? Also, the link to your github from your site goes to stdbrouw instead of debrouwere
- stdbrouw 12y ago
  So do you feel the pitfalls of probabilistic programming / Bayesian data analysis are potentially more severe than those of traditional methods? I don't get to use that much Bayesian analysis in my work, but I've been toying around with a hierarchical model of content popularity recently: are some topics / authors / genres more popular than others? Frankly finding it hard to separate the signal ("this is what people actually like to read") from the noise ("this is what just happened to be trending and will not be reproducible").