4 ms·

This is a very old argument. I got my bachelor's degree in psychology at Harvard in 1993, and was told repeatedly that p-tests are abused, overused, and not ter

This is a very old argument. I got my bachelor's degree in psychology at Harvard in 1993, and was told repeatedly that p-tests are abused, overused, and not terribly useful. To my mind, the most hackable flaw is that the number of subjects in the study is a term in the denominator of the p-value calculation. Any study with a sufficiently large sample will find "significance" with p<.05, but it won't be meaningful. We were taught that "effect size" measures were crucial to understanding and interpreting results. If you have p<.05 but a tiny effect size, you're likely not seeing a meaningful difference.

DataWorker 7y ago
Odd that in some ways psychology seems to be more informed, as a field, of these nuances and pitfalls and yet they have failed as a field to adequately police themselves. The problem isn’t on the statistics end of things, it’s the labor market and gatekeeping end of things. Another aspect of the inability to scale higher ed in the stupid way that many think is optimal.
JamesBarney 7y ago
> To my mind, the most hackable flaw is that the number of subjects in the study is a term in the denominator of the p-value calculation. Any study with a sufficiently large sample will find "significance" with p<.05, but it won't be meaningful. Just to clarify. Any study with a sufficiently large sample will find a real effect even if the effect isn't clinically significant.
bjourne 7y ago
> To my mind, the most hackable flaw is that the number of subjects in the study is a term in the denominator of the p-value calculation. Any study with a sufficiently large sample will find "significance" with p<.05, but it won't be meaningful. I don't think that is right. Think about tossing unbiased coins. The probability that you will get an improbable (p < 5%) result is exactly the same no matter how many or few coins you toss.

DataWorker 7y ago

Odd that in some ways psychology seems to be more informed, as a field, of these nuances and pitfalls and yet they have failed as a field to adequately police themselves. The problem isn’t on the statistics end of things, it’s the labor market and gatekeeping end of things. Another aspect of the inability to scale higher ed in the stupid way that many think is optimal.

JamesBarney 7y ago

> To my mind, the most hackable flaw is that the number of subjects in the study is a term in the denominator of the p-value calculation. Any study with a sufficiently large sample will find "significance" with p<.05, but it won't be meaningful. Just to clarify. Any study with a sufficiently large sample will find a real effect even if the effect isn't clinically significant.

bjourne 7y ago

> To my mind, the most hackable flaw is that the number of subjects in the study is a term in the denominator of the p-value calculation. Any study with a sufficiently large sample will find "significance" with p<.05, but it won't be meaningful. I don't think that is right. Think about tossing unbiased coins. The probability that you will get an improbable (p < 5%) result is exactly the same no matter how many or few coins you toss.