Tangential point: Because of frictions and biases, the need for high-power and replicated research is even larger if you want to translate the science into real

Tangential point: Because of frictions and biases, the need for high-power and replicated research is even larger if you want to translate the science into real-world action. I highly recommend this post by Holden Karnofsky of GiveWell (and formerly of Bridgewater): http://blog.givewell.org/2016/01/19/the-importance-of-gold-standard-studies-for-consumers-of-research/ http://blog.givewell.org/2016/01/19/the-importance-of-gold-s... GiveWell is an organization that tries to estimate the impact per philanthropic dollar of various charities, and they partially rely on controlled trials of health interventions in the developing world. > Chris Blattman worries that there is too much of a tendency toward large, expensive, perfectionist studies, writing: "...each study is like a lamp post. We might want to have a few smaller lamp posts illuminating our path, rather than the world’s largest and most awesome lamp post illuminating just one spot. I worried that our striving for perfect, overachieving studies could make our world darker on average." > My feeling – shared by most of the staff I’ve discussed this with – is that the trend toward “perfect, overachieving studies” is a good thing... > Bottom line. Under the status quo, I get very little value out of literatures that have large numbers of flawed studies – because I tend to suspect the flaws of running in the same direction. On a given research question, I tend to base my view on the very best, most expensive, most “perfectionist” studies, because I expect these studies to be the most fair and the most scrutinized, and I think focusing on them leaves me in better position than trying to understand all the subtleties of a large number of flawed studies. > If there were more diversity of research methods, I’d worry less about pervasive and correlated selection bias. If I trusted academics to be unbiased, I would feel better about looking at the overall picture presented by a large number of imperfect studies. If I had the time to understand all the nuances of every study, I’d be able to make more use of large and flawed literatures. And if all of these issues were less concerning to me, I’d be more interested in moving beyond a focus on internal validity to broader investigations of external validity. But as things are, I tend to get more value out of the 1-5 best studies on a subject than out of all others combined, and I wish that perfectionist approaches were much more dominant than they currently are.