6 ms·
Every researcher would love to include error bars but it's a matter of limited computing resources at universities. Unless you're training on a tiny dataset lik
by zone411 4y ago
Every researcher would love to include error bars but it's a matter of limited computing resources at universities. Unless you're training on a tiny dataset like MNIST, these training runs get expensive. Also, unless you parallelize from the start and risk wasting a lot of resources if something goes wrong, it could take longer to get the results.
- PaulHoule 4y agoUsing bootstrap and/or repeated runs is a great way to get error bars but there are low cost ways to do it. For instance they estimate error bars on public opinion polls based on simple formulas and not redoing the poll a large number of times.
- nequo 4y agoIf you don't have an analytical expression for your asymptotic variance, you do have to use bootstrap though. For public opinion polls, the estimator is simple (i.e., a sample mean), so we have an analytical expression for its asymptotic variance.
- time_to_smile 4y agoSimple formulas only work because the models themselves for those polls are incredibly simple and adding a bit more complexity requires a lot of tools to compute these uncertainties (this is part of the reason you see probabilistic programming so popular for people doing non-trivial polling work). There are no simple approximations for a range of even slightly complex models. Even some nice computational tricks like the Laplace approximation don't work on models with high numbers of parameters (since you need to compute the diagonal of the Hessian). A good overview of the situation is covered in Efron & Hastie's "Computer Age Statistical Inference".
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]