3 ms·
The problem with a fixed criteria of p < 0.05 for statistically significant results is that it is only useful if each experiment measures only one thing and all
by gus_massa 8y ago
The problem with a fixed criteria of p < 0.05 for statistically significant results is that it is only useful if each experiment measures only one thing and all the studies are published.
I remember an study of a few year ago where they had
* rats and mice -> x2
* male/female/both -> x2 (or x3)
* three level of radiation -> x3
* two or three types of cancer -> x2 (or x3)
So the report has at least 2x2x3x2=24 results. There is a high probability that by chance one of them gets a p<0.05.
Also, studies with a positive result have a higher chance of been published, because the researchers are not interested in publishing boring studies without a positive result so they don't write it, and the journals are not interested in publishing boring studies without a positive result so they don't approve it. (And I'm ignoring intentional fraud and p hacking ...)
Anyway, perhaps the researchers sponsored by the telephone companies/manufacters just publish all their studies and the independent researches publish only the interesting studies. This is why in certain areas it is mandatory to register the experiments before they start, to ensure (reduce) there is no publication bias.
In particle physics it's too easy to do experiments, filtering and analysis, so they use a criteria of 5 sigmas to call it a discovery. This is roughly p<0.0000003, because if they use p<0.05 you would get every month a announcement of a new particle and a few month later the retraction.
For experiments with mice/rats it's impossible to ask for p<0.0000003 because you will need trillions of rats, but remember that p<0.05 is just an arbitrary threshold.