6 ms·
Why is 0.05 so common to begin with. Its an arbitrary value ain't it?
by kalium-xyz 7y ago
Why is 0.05 so common to begin with. Its an arbitrary value ain't it?
- paulddraper 7y agoIt's as arbitrary as the age 18. Not a magic number, just a threshold with consensus.
- kazinator 7y agoIf there is consensus, why are we seeing calls for abolishment from the scientific community?
- probably_wrong 7y agoI think you might be conflating two related-but-different arguments. There is consensus that a p-value of 0.95 is reasonable. No one is arguing against that parameter in particular. The argument is rather against the idea of statistical significance itself, because it's relatively easy to cheat if you are dishonest. Setting p to 0.01 won't change that. Finally, and this is my personal opinion, there is "consensus" for abolishing statistical significance the same way there's "consensus" that Python is the best programming language. Scientists are more or less in agreement that it could be better, but "mildly displeased scientists call for unclear improvement" is not as good a headline.
- deleted 7y ago[deleted]
- 6gvONxR4sf7o 7y agoThere is not consensus that the current threshold is reasonable. There are many big names arguing against that parameter in particular. See e.g. Redefine statistical significance. The authors list is practically a who's who https://psyarxiv.com/mky9j https://psyarxiv.com/mky9j >One Sentence Summary: We propose to change the default P-value threshold for statistical significance for claims of new discoveries from 0.05 to 0.005.
- Insanity 7y agoIsn't consensus essentially a majority agreement, and thus not a complete agreement? Furthermore, a consensus today does not need to hold up indefinitely.
- paulddraper 7y agoSame with the age 18 :) Historical consensus, then.
- lisper 7y agoYep. But if you're going to make binary publishing decisions then you have to draw a line somewhere.
- kazinator 7y agoHowever "1/20 chance that, if the hypothesis is false, our research confirms it anyway" is not a good choice of somewhere for any important research that can affect people's lives.
- anbende 7y agoWhat would you suggest, and what contexts do you think .05 would be appropriate for, if any?
- feral 7y agoThis debate isn't that simple. The other side of this isn't that everything magically just gets more accurate. The cost will be withholding true discoveries from the medical community for longer. Imagine you've done a study of something else and you accidentally discover a (properly FDR controlled) correlation between some drug and heart attacks, significant at 0.03 and with no big contrary prior. Should you publish or wait 5 years for a follow-up study to complete? What if three different groups stumble across this same finding? Should they all publish and maybe someone will realize 'shit this drug is killing people' or should they all wait to hit some other higher standard of significance? The point is theres always a balance between specificity and sensitivity, a tradeoff in terms of costs. I'd personally be happy with keeping 0.05 as the threshold for 'probably something here'. The real issue are about publication bias, incentives, naive interpretation of published work 'heres one study in Nature so it must be true'. I don't see any other purely statistical change, Bayesian (essentially has all the same problems, aside from taking into account priors) or whatever, that will solve these in any way that won't come without unacceptable sensitivity cost.
- 6gvONxR4sf7o 7y agoThere have been some recent proposals to lower it getting lots of traction. Sure if you want to decide whether to publish based on p-value alone you have to draw a line somewhere, but 0.05 isn't necessarily the right place. And you don't make publishing decisions based on p-values alone.