4 ms·
More like "make sure your test power is what you think it is". There will still be results that fail to replicate by virtue of the rejection of the null hypothe
by hackerlight 4y ago
More like "make sure your test power is what you think it is". There will still be results that fail to replicate by virtue of the rejection of the null hypothesis by chance, but that should only happen 1 in 20 times at an alpha of 0.05. With all the bad practices that alter test power, such as p-hacking and the file drawer effect, that 1 in 20 blows up to 1 in 2.
- hackandthink 4y agoGelman makes a distinction between p-hacking and choosing the test based on your data. p-hacking is confirming your preferred hypothesis - just try again. the latter let's you write a paper. (test power is different: if there's really an effect, would you detect it)
- hackerlight 4y agoOops, I meant to say test size, not test power. A test size of 0.05 is only really 0.05 if the entire process you used was free from p-hacking. And as a reader of that paper that process includes the file drawer effect. All these things have an impact on distorting the test size away from what it is advertised to be.