5 ms·
This is interesting, but these sorts of "reproducibility initiatives" also kind of naive: there's a natural meta-review process in science, wherein research ge
by timr 14y ago
This is interesting, but these sorts of "reproducibility initiatives" also kind of naive: there's a natural meta-review process in science, wherein research generally isn't accepted until it has been reproduced and published in different ways by multiple labs. In other words, peer-review of individual papers is not the only sanity check (or even the most important sanity check).
That's not to say that these sorts of things aren't useful exercises, just that it's pretty wishful thinking to argue that this sort of peer-replication is going to make any sort of dent in the system. Scientists are a naturally skeptical and conservative group of people, and one of the first lessons that gets drilled into you in graduate school is that you have to take individual papers with huge grains of salt.
- akiselev 14y agohttp://blogs.nature.com/news/2011/09/reliability_of_new_drug_target.html http://blogs.nature.com/news/2011/09/reliability_of_new_drug... The problem is that there is a lot of incentive to publish peer reviewed articles and very little incentive to publish articles confirming another paper. In my experience, the reproducibility of papers is tested in the worst possible way: during the process of more novel research. If we are to continue this exponential growth in technological and scientific progress we need to set up a far more efficient way of reproducing the papers, independently of ongoing novel research. This seems like a great way, especially considering the benefit it will give to startups and other companies that can allow new R&D to rely on papers that have been tested outside of the peer review/slow meta-review process in science.
- timr 14y agoIt depends on how you look it it: is it "more efficient" to spend the time and energy independently reproducing every paper that comes down the pipe, or to wait for researchers verify the work that actually matters? I think this kind of thing is probably a well-intentioned pipe dream. You can't predict which work is going to be important, so you can't predict, a priori, which work to verify. So you either have to verify all of it, or you have to guess, or...you can just let people do what they do right now: verify the work they care about.
- akiselev 14y agoWell for one they don't HAVE to reproduce every paper. They only have to reproduce protocols proposed for these papers, and mostly you can stick to the ones with obviously wide implications. If, for example, there was one group who tested all of the methods for manufacturing graphene for efficiency and repeatability, that would make work much easier for dozens if not hundreds of labs around the world. Not to mention it would open up the way for startups, who can rely on tested protocols, to begin industrial scale work with graphene. You can't predict which work would be important in general, yes, but when it comes to identifying important protocols in organic chemistry and biology, there is a lot more information on what is important and what is not. Most of the time, in o. chem, protocols are incremental improvements over previous protocols and it's very easy to figure out which of them are most often used.
- james_ash 14y agoLike I alluded to in a previous comment, in this particular field the experimental feedback loop is short (often <24h). This type of peer-replication has led to the retraction of at least one paper in a highly prominent journal (JACS): http://totallysynthetic.com/blog/?p=1903 http://totallysynthetic.com/blog/?p=1903 . So the potential to make a dent is there, for this field at least.