3 ms·
>If p=0.05 is sufficient confidence to publish, one in twenty published findings will be wrong This is a wrong understanding of p values. Imagine an urn with o
by Sinidir 5y ago
>If p=0.05 is sufficient confidence to publish, one in twenty published findings will be wrong
This is a wrong understanding of p values. Imagine an urn with only fair 50/50 coins and taking out coins to test them. If you reject the coin being fair at 0.05<p then you will on average have 1/20 statistically significant results. All of your statistically significant results will be wrong.
Now imagine an urn that only has biased coins.
Every time you reject the null hypothesis you are correct.
How many statistically significant results are actually true
depend on the metaphorical urn that people draw hypothesis out of.
If someone is very good at picking likely hypothesis to pan out their statistically significant results are more likely to be true.
Also if a field of study has many bad paths you could go down on then you would expect more statistically significant results to be actually wrong.
- marginalia_nu 5y agoLet's not pretend that the practice of throwing a bunch of models at a dataset in search of statistical significance isn't a thing. P-hacking is a pretty big problem in some fields. That's one of those context-things I mentioned. It's easy to go from a field where you can control variables very well, like physics, and look at a field where the variables are all over the map, e.g. psychology or nutrition; and expect the contexts to be similar because science is science. But it's not. The latter fields have really big problems actually replicating published results, many times the findings just never materialize outside of the initial study. It's not all from overt bad science like p-hacking, effects like publication bias also appears to play a big role.