6 ms·
>For a wide range of common statistical tests, transitioning from a P value threshold of α = 0.05 to α = 0.005 while maintaining 80% power would require an in
by imh 9y ago
>For a wide range of common statistical tests, transitioning from a P value threshold of α = 0.05 to α = 0.005 while maintaining 80% power would require an increase in sample sizes of about 70%.
This proposal is a great pragmatic step forward. Like they say in the paper, it doesn't solve all problems, but it would be an improvement with reasonable cost and tremendous benefits.
>Such an increase means that fewer studies can be conducted using current experimental designs and budgets. But Fig. 2 shows the benefit: false positive rates would typically fall by factors greater than two. Hence, considerable resources would be saved by not performing future studies based on false premises.
- stdbrouw 9y agoIn some fields like psychology, power is more likely to already be 10% or 20% for the majority of studies, and in fact P-hacking and low standards for evidence would be far less harmful if power were higher, because low power leads to inflated effect size estimates. Additionally, power calculations are always just a guess and easy to fudge, so it's pretty much a given that current statistical power would not be maintained with more stringent critical values. See http://andrewgelman.com/2014/11/17/power-06-looks-like-get-used/ http://andrewgelman.com/2014/11/17/power-06-looks-like-get-u... So this proposal is really the opposite of pragmatic. Pragmatic would be requiring effect size estimates and confidence intervals in all published papers. It is surprising how many papers will talk about highly significant effects without actually discussing how large the estimated effect is thought to be, which gives authors a lot of leeway when exaggerating the importance of their findings.
- reilly3000 9y agoThe cost of increasing sample size is significant; this is a trade-off that allows smaller projects to still conduct valuable research.
- Klockan 9y agoStudies with low significance are no better than anecdotes.
- stdbrouw 9y agoThis is not true at all. Small studies are simply less accurate. Would you prefer to conduct further research on a cure for cancer that's been shown to lead to recovery in 0-50% of cases (the confidence interval includes 0%, so not significant) or in .5-1% of cases (significant)?
- Will_Parker 9y agoHow does it make sense to suppose it could lead to recovery in 50% of cases, if a drug that does nothing has a 5% chance of producing the same result in your study?
- stdbrouw 9y agoBecause it has a 95% chance of producing a different (better) result. Statistics is all about the quantification of uncertainty.
- imh 9y agoAs a matter of fact, I can do even better than 0-50% of cases! Consuming widgets has been shown to lead to recovery in 0-100% of cases! It isn't as simple as you suggest.
- stdbrouw 9y agoYep, and that makes widgets a more interesting target for follow-up studies than a treatment that has been shown to result in no clinically significant improvement. It really is that simple.
- Klockan 9y agoSurely we could create a system to fund extensions of studies with promising initial results? I just think that it is dangerous to call 95% significant.
- maxerickson 9y agoSo ultimately the issue is that push button statistics don't work?
- xaa 9y agoUltimately the issue is that we somehow got fixated on the p-value, which (roughly and imprecisely speaking) quantifies the probability that there is any effect, even a small one, rather than using effect size estimates which estimate both the magnitude of the observed effect and the uncertainty in that estimate. Using p-values as our primary metric means an overemphasis on finding small effects (which are usually not clinically relevant anyway) and unduly low focus on things with big effects. If an effect is real, but very small, that too may well cause replicability problems because it suggests the effect may not be very robust to small changes in experimental conditions, whereas a big effect would be more likely to be robust. If you think about the really important scientific findings -- the ones that made a big impact and are indisputably true -- statistics usually aren't necessary to prove them, because the effect size is so large it is simply obvious. I'm not against using statistics anyway, of course, but the point is that we should be looking mainly for effects with big effect sizes if we are after important findings, IMO. It is only a major bonus that big effect sizes are most likely to be replicable.
- autokad 9y agoalmost all problems has to do with data and selection of data. changing the p-value threshold wouldn't help.
- coverband 9y agoI upvoted you in principle, but working with a tighter threshold would also make choosing self-serving data samples more obvious, if not more difficult.
- scottlocklin 9y agoThe real solution is social scientists using statistics properly. Data mining 0.005 p-values isn't practically speaking much harder than 0.05. And some of these clowns are actually data mining on purpose; I've listened to them talking about it.