3 ms·

This is a more mathematically precise restatement of kerkeslager's explanation below and the obligatory xkcd [1]. The problem boils down to the fact that null r

This is a more mathematically precise restatement of kerkeslager's explanation below and the obligatory xkcd [1]. The problem boils down to the fact that null results are not usually published, and that intuitive skepticism is not quantifiable. Essentially, we want to get p(H0|x), but we need Bayes Law to get this from p(x|H0). But we need some notion of what priors to use. This is of course impossible to actually get, but if we published null studies then it would allow us to estimate it with better confidence. Alternatively we can ballpark it by saying how unexpected the result is, and how long the effect being explained has been studied. The "common-sense" version of this is intuitive, that "extraordinary claims require extraordinary evidence". The more unexpected a result is, the lower the p-value has to be to be convincing. The epidemiological version of this shows up in things like the Bradford Hill criteria [2], which include significance of association but also attempt to bring in plausibility. [1] https://xkcd.com/882/ https://xkcd.com/882/ [2] https://en.wikipedia.org/wiki/Bradford_Hill_criteria https://en.wikipedia.org/wiki/Bradford_Hill_criteria

CreRecombinase 7y ago
I'd argue that p(H0|x) is also (in most cases) pretty uninteresting from a scientific perspective, and so the whole "publish all the p-values" solution I think is only fighting half the battle. Gelman does a much better job of arguing this than I ever could [1], but this idea that rejecting the null is the "desired" outcome is the real problem. To put it another way, a low p-value is another way of saying "my model of the data generating process is bad", which is in most cases a pretty unsatisfactory outcome. As to the idea that priors are "impossible" to get, to quote Gelman [2] again, why "strain at the gnat that is the prior and swallow the ungainly camel that is the iid likelihood? [1] https://statmodeling.stat.columbia.edu/2015/03/02/what-hypothesis-testing-is-all-about-hint-its-not-what-you-think/ https://statmodeling.stat.columbia.edu/2015/03/02/what-hypot... [2] https://statmodeling.stat.columbia.edu/2015/07/03/why-should-anyone-believe-that-why-does-it-make-sense-to-model-a-series-of-astronomical-events-as-though-they-were-spins-of-a-roulette-wheel-in-vegas/ https://statmodeling.stat.columbia.edu/2015/07/03/why-should...

CreRecombinase 7y ago

I'd argue that p(H0|x) is also (in most cases) pretty uninteresting from a scientific perspective, and so the whole "publish all the p-values" solution I think is only fighting half the battle. Gelman does a much better job of arguing this than I ever could [1], but this idea that rejecting the null is the "desired" outcome is the real problem. To put it another way, a low p-value is another way of saying "my model of the data generating process is bad", which is in most cases a pretty unsatisfactory outcome. As to the idea that priors are "impossible" to get, to quote Gelman [2] again, why "strain at the gnat that is the prior and swallow the ungainly camel that is the iid likelihood? [1] https://statmodeling.stat.columbia.edu/2015/03/02/what-hypothesis-testing-is-all-about-hint-its-not-what-you-think/ https://statmodeling.stat.columbia.edu/2015/03/02/what-hypot... [2] https://statmodeling.stat.columbia.edu/2015/07/03/why-should-anyone-believe-that-why-does-it-make-sense-to-model-a-series-of-astronomical-events-as-though-they-were-spins-of-a-roulette-wheel-in-vegas/ https://statmodeling.stat.columbia.edu/2015/07/03/why-should...