4 ms·

It would be interesting to know how reproducible the reproducibility test turns out to be. Edit: Just to expand a bit -- Suppose I flip a fair coin 100 times

It would be interesting to know how reproducible the reproducibility test turns out to be. Edit: Just to expand a bit -- Suppose I flip a fair coin 100 times, and publish each result as its own god-given Truth. Now somebody comes along and questions my 100 Truths, so they flip a coin 100 times; The expectation value is that 50 of my Truths reproduce. Naturally there is a sigma (of about 10 I think), so the reproducibility study shows 50 ± 10. The linked reproducibility study is right around expection - sigma. I realize that's numerology, but kinda funny. Nevertheless, my original point: The reproducibility study should be run several times to understand the random nature of the results. (Do the same studies reproduce their results? Or is it a different 50-ish results?)

capnrefsmmat 12y ago
What you're asking is basically the concept of statistical power. Assume the original study found an effect size E, and we take that as the truth. How likely is the replication attempt to find a statistically significant effect? The Reproducibility Project calculated the sample sizes necessary in advance, so if the effects are the size the original researchers claim, they'd have good power to detect them. Their power could be worse than they expect, though; pioneering studies tend to overestimate effect sizes, because their sample sizes are too small and they filter for statistical significance. I call the problem truth inflation: http://www.statisticsdonewrong.com/regression.html#truth-inflation http://www.statisticsdonewrong.com/regression.html#truth-inf... Anyway: understanding the random nature of results is exactly the job of statistics, and the Reproducibility Project researchers are being very careful with their statistics.