3 ms·
Bayesian modeling can be very powerful when it works but it can also be catastrophic when it fails. It helps to think about this in an adversarial decision the
by howlin 7y ago
Bayesian modeling can be very powerful when it works but it can also be catastrophic when it fails. It helps to think about this in an adversarial decision theoretic context where you play a prediction game against an opponent (usually called Nature).
We can think of the game as discovering the best model to explain a set of observations. The Bayesian believes that Nature picks the true model that generated the observations by sampling the prior. This is actually a huge assumption to make, which is why Bayesian methods work so well when the assumption is close to the truth.
Frequentists make the assumption that Nature chooses the underlying true model from a set of possible models. Beyond restricting the set of models Nature can choose from, frequentists make no further assumptions about the selection process. This is a strictly weaker assumption than the Bayesian makes, which means frequentist methods will do better when the specified prior grossly misrepresents Nature's decision making process.
There are even weaker assumptions that can be made about how Nature chooses the data. Regret-based model inference allows for a more adversarial game with Nature where the data may not come from the class of models considered at all. If Nature truly behaves this way, then Bayesian decision making can catastrophically fail.
- c2471 7y agoThis ignores the main strength of a Bayesian workflow. You can straight forwardly quantify the effect of your prior choice on your inference - pick a different prior; how much does that change the inference, etc etc. A good Bayesian workflow does not assume a prior to be true; it should be based on available evidence, and then stressed. To be a bit more concrete, let's say we wish to model the height of kangaroos. We come up with a model form, say regression, and a bunch of potential features. If we are Bayesian we might say; "I think nature prefers simple stable solutions, so I'll put a N(0,d) prior on my weights. We then compute a posterior and get a range of credible values. We can then say, "hey, what if I'm wrong and actually it's a student t, or it's flat prior or X or y or z", and use principled tools like marginal likelihood to say which family of models works best, do prior posterior comparisons to see how observations changed our prior etc etc. If we do this under a frequentist framework we compute the regression coefficients, and can get some confidence bounds with some appeal to asymptotics (and nobody I've ever seen actually makes any attempt to validate these assumptions). And even when we are done, we get a confidence interval that has such a truly unintuitive definition that almost every person who is not a stats PhD fundamentally misinterprets. To say frequentists make less assumptions is not true- they are just less explicit, and I consider it a strength not a weakness to highlight choices made by the statistician.
- nazgulnarsil 7y agoRight, one should run a sensitivity analysis in general, and your prior is one of the parameters you definitely check the sensitivity of.
- analog31 7y agoAs a thought experiment, could you choose priors by setting the derivative of the solution with respect to the priors equal to zero? This would be the case of minimal sensitivity.