3 ms·
Yeah - this is a sample of the 'gold standard' of psychology papers, and only 10/13 could be reproduced. The reasons for shoddy reproducibility are p-value hac
by Fede_V 12y ago
Yeah - this is a sample of the 'gold standard' of psychology papers, and only 10/13 could be reproduced.
The reasons for shoddy reproducibility are p-value hacking, intense pressure to publish at all costs, and a premium on 'gladwellesque' results where a simple theory seemingly explains a lot.
Gellman and Uri Simon-Johnson have both written a lot about this.
- lsiebert 12y agoConsidering that it included a larger and more diverse sample across 12 countries, it may be that the original studies which couldn't be supported were valid but not widely applicable beyond a specific population, or that the results represent a change in how people act from the time of the studies (One of them was an effect based on seeing an American Flag, and I can see how that might have changed).
- darkxanthos 12y agoThe original paper considers this and finds that to be of limited if not negligible concern. The experiment designs were extremely meticulous and even included the original authors whenever possible.
- lsiebert 12y agoGreat point!
- ekidd 12y agoYeah - this is a sample of the 'gold standard' of psychology papers, and only 10/13 could be reproduced. Having watched scientists at work, reproducing 10 of 13 high-profile studies sounds pretty reasonable. Given p≤0.05, you'd expect 19/20 to be reproducible. Then you add in other factors: - Surprising positive results get published more readily than negative ones. - Some results may be sensitive to tiny changes in methodology. - And more: "Why Most Published Research Findings are False" http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/ http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/ Some of these effects can be reduced with pre-registration of studies, and with other methodological improvements. But in general, even when everybody plays by the rules, a fair bit of garbage is going to slip through, because it's hard to eliminate all sources of error and bias. So when a scientific field can say, "Hey, 3/4ths of our really interesting results are real!", that's about what I expect when the processing is working.
- stdbrouw 12y agoAnd another big one: the low statistical power of much psychological research, meaning that even if an effect is present, the original study or its attempted replication might not find it due to inadequate sample size.
- capnrefsmmat 12y ago> Given p≤0.05, you'd expect 19/20 to be reproducible. No, that's not the case; follow your link to "Why Most Published Research Findings are False" to see why that's not the case. The "positive predictive value" of a discovery, even without any bias, is very different from what the p value would imply.
- ekidd 12y agoYes, this is an excellent point. To take an extreme example, if there are no true, interesting results to be discovered in a field of research, then any study claiming such a result is by definition false. Analogously, if none of your employees do drugs, all positive drug tests are false positives.
- jpmattia 12y ago> Yeah - this is a sample of the 'gold standard' of psychology papers, and only 10/13 could be reproduced. It also raises the question of whether the reproducibility study is itself reproducible.