3 ms·

This would also require a reduction in the "publish or perish" pressure, because it would significantly lengthen the time from study design to publishing. Furt

This would also require a reduction in the "publish or perish" pressure, because it would significantly lengthen the time from study design to publishing. Furthermore, who's going to be doing that independent reproduction? How are you going to incentivize that? If you require independent reproduction for every paper in a field where that's even possible, that's going to mean either you need to double the rate of running experiments in that field (unlikely), or double the number of researchers in that field (also unlikely) if you want to maintain anywhere close to the current rate of papers being published. (And again, even if you could double the number of researchers, you still need to provide enough of an incentive to reproduce that literally everyone is doing so for half of the studies they run. And you'd somehow have to ensure that you don't end up with fun, interesting studies getting 15 different groups reproducing them, while less-flashy studies that are still important fundamental science languish for years without anyone caring to try.) Given how much of "publish or perish" is institutional inertia and culture, the idea that it could be systemically "nudged" to allow for this kind of shift is essentially a fantasy. Far too many of the Old Guard who are already in charge of the tenure committees would say, "Well, I had to publish 327 papers to be given tenure, and kids these days have so many newfangled things that let them write papers faster; why shouldn't we require them to publish 327 papers, too??"

notahacker 3y ago
To add to your "who's going to be doing that independent reproduction?" point, the whole purpose of the journals is to communicate to the rest of the scientific community that there's an interesting result here that might want further examination. Sure, in the absence of the ability to disseminate your findings via the usual channel, you might be able to instruct your grad students to reproduce or beg one of your regular collaborators to reproduce, but then the independence of the experiments is dubious. And otherwise, academics aren't exactly searching for unpublished papers to replicate so the original can be published.
anonymouskimmer 3y ago
> If you require independent reproduction for every paper in a field where that's even possible, that's going to mean either you need to double the rate of running experiments in that field (unlikely), or double the number of researchers in that field (also unlikely) You don't have to quite double the number of researchers or experiments. It typically takes less work to replicate final results than all of the work getting to the point of good results. Presumably a lot of the replication would be outsourced to CROs. Though since the CROs would be focused on replication of results alone, any theoretical errors or errors in experiment design wouldn't be discovered. Replicating results will mostly catch fabricators. It won't catch nearly all of the far more important to the replication crisis bad experimental design.

notahacker 3y ago

To add to your "who's going to be doing that independent reproduction?" point, the whole purpose of the journals is to communicate to the rest of the scientific community that there's an interesting result here that might want further examination. Sure, in the absence of the ability to disseminate your findings via the usual channel, you might be able to instruct your grad students to reproduce or beg one of your regular collaborators to reproduce, but then the independence of the experiments is dubious. And otherwise, academics aren't exactly searching for unpublished papers to replicate so the original can be published.

anonymouskimmer 3y ago

> If you require independent reproduction for every paper in a field where that's even possible, that's going to mean either you need to double the rate of running experiments in that field (unlikely), or double the number of researchers in that field (also unlikely) You don't have to quite double the number of researchers or experiments. It typically takes less work to replicate final results than all of the work getting to the point of good results. Presumably a lot of the replication would be outsourced to CROs. Though since the CROs would be focused on replication of results alone, any theoretical errors or errors in experiment design wouldn't be discovered. Replicating results will mostly catch fabricators. It won't catch nearly all of the far more important to the replication crisis bad experimental design.