3 ms·
It is actually much much worse. Let's do the math. 1,000 non-tenured professors do 20 experiments each. On average, each of them has one p<0.05 result that is
by beagle3 12y ago
It is actually much much worse. Let's do the math.
1,000 non-tenured professors do 20 experiments each. On average, each of them has one p<0.05 result that is actually a coincidence. They all rush through publication in an attempt to get tenure.
Unless you have reason to believe some results are actually plausible, it's possible ALL published results are wrong.
The 5% is the ratio between "experiments done" and "results that are not meaningful yet randomly deemed worthy of publication". It has no relation to the number of "results that are actually meaningful", which may outnumber the former 100:1 ... or, more likely, be outnumbered by them 100:1.
I wish Neyman/Pearson theory was taught more widely, or at least its implication to the false-positive/false-negative tradeoff. Alas, that would mean a lot less published results and a lot less bragging/tenure rights.