A cynical view: the job of a p-value is to make social scientists and psychologists look like they have great publication records. I sometimes think this should

A cynical view: the job of a p-value is to make social scientists and psychologists look like they have great publication records. I sometimes think this should be set as a compulsory first exam question in any large-cohort p-value-using discipline: Bob has a class of 100 undergraduate students. He tells them to go and run one experiment each as part of their final year class. Assuming all the data in all the experiments is completely random, what is the probability that at least one student will nonetheless find a "statistically significant effect" (p < 0.05) that can then be written up and get published. rough answer: 99.4% p < 0.05 seems to be chosen as that's a barrier that is not too harsh on the individual researcher (at very high significance levels it would be hard for anyone to run a powerful enough study in order to get published and keep their research job), but p-values are essentially defeated as a filter for genuine effects by the sheer number of experiments being run around the world. In some senses, the answer here is simply to stop brandishing science as a social cudgel ("X is science; how dare you not believe it" or more often "how dare you employ/fund someone who does not believe it, they must be sacked") which puts it on a pedestal of being canonical truth all the time. It is only the fact that science has recently been used more and more as a political stick to beat opponents with (and no, not just in climate and evolution, but right down to things like the best way to teach reading, or whether schools should be regulated or independent) that has meant that people think "science is broken" whenever it turns out to have got a wrong result. No, it's just expected to get it wrong fairly often. And in practice the most common "self-correction" process in science isn't a repeated experiment and published retraction, but just academics reading the paper, thinking "this one's garbage", and not basing their research from it. Publication is not strictly a test of truth -- just a test of methodology and analysis. The test of truth always occurs in the mind of the reader.