3 ms·
> This is a problem related to A Priori Probability, where you're essentially turning research, which could have been statistically applied to a larger populati
by p4wnc6 11y ago
> This is a problem related to A Priori Probability, where you're essentially turning research, which could have been statistically applied to a larger population given some statistical confidence, into deductive reasoning which doesn't necessarily apply beyond the finite sample size [1] (also see @entee's comment and links for better explanation of why this is important).
Maybe you are using these terms differently than they are used professionally, but this is not correct if it is meant as a criticism of Bayesian methods.
There is nothing inherently better (and in fact there are several things potentially worse) about looking at the data with inductive reasoning about a potentially larger population with assumed baseline properties.
The problem here is simply p-hacking, whether by accident or intentional. This can be done whether you are performing Bayesian analysis involving a prior distribution, or performing frequentist analysis with assumptions about null hypothesis and subjectively chosen rejection criteria.
If you don't register your model validation procedures ahead of time, then after the fact you can data mine among many (or even all possible) model validation scores and only publish scores that show a positive effect.
It's not about one framework of statistics or the other. It's about statistical rigor for whatever framework. It's about pursuing more than one study. And it's about attempting to use holistic model validation techniques that cover wide ranges of anticipated outcomes, to hopefully drive down the chances that you develop a conclusion from the study based only upon a narrow set of outcomes that could be biased.
- JangoSteve 11y agoI am definitely not using the terms as they are used professionally (and certainly not a criticism of Bayesian methods in any way). It was an attempt to explain in more lay terms how when p-hacking is performed on some dataset, the "findings" can rarely be applied to larger (or different) datasets due to the arbitrary criteria used to reject the parts of the data that aren't conducive to the selected outcome, since the rejection criteria were sort of custom-fit to that specific dataset. I agree with everything you said.