4 ms·
Does anyone know what a practical upper limit is for Stan in terms of size of data set / number of parameters to fit? Can you use stan to fit a model with ~10^4
by elsherbini 6y ago
Does anyone know what a practical upper limit is for Stan in terms of size of data set / number of parameters to fit? Can you use stan to fit a model with ~10^4 parameters and ~10^6 rows of data if you had access to ~10^3 cores? How long would it take?
- gbrown 6y agoI imagine it depends on which algorithm you’re using - maybe with their VB functionality (I almost entirely use the souped up NUTS algorithm for full Bayesian inference). It also likely depends on how well conditioned your model is - even if you can get it to run for huge models on reasonable hardware, convergence may not be practical.
- hendzen 6y agoStan used to only be able to parallelize across chains but they introduced within-chain parallelism this year. Even then some of the work is still serial so I don’t think you can expect linear speedups past a certain point.
- standevbob 6y agoYes, we regularly use Stan's MCMC to fit relatively simple time-series regression models or item-response theory type models with 10^5 parameters and 10^6 rows of data on a desktop computer. It can take a day, though. It's much faster with variational inference, but that can be less stable and it doesn't give you the same uncertainty quantification because of the way the KL-divergence is ordered in the objective. Stan can parallelize multiple chains and it can parallelize the density/gradient calculations in a single chain. But for the latter to be efficient, the chunks being parallelized need to be compute intensive, like you might get in a pharmacometric compartment model where you might have to solve a bunch of differential equations for each of thousands of patients in a clinical trial.