3 ms·

There's an interesting asymmetry in peer review. If I invite an outside scientist to review my work once it is complete, that's external peer review, and it con

There's an interesting asymmetry in peer review. If I invite an outside scientist to review my work once it is complete, that's external peer review, and it confirms that my work is sound. If I invite the outside scientist to collaborate on the work from the beginning, so they can catch mistakes before they ruin the work, that's not peer review -- I'd have to go out for peer review at the end. Why the focus on catching errors when they've already been committed? Some journals (https://osf.io/8mpji/wiki/home/ https://osf.io/8mpji/wiki/home/) have started "registered reports", where you send your methodology out for peer review before running the experiment, and the journal commits to publishing the results if the methodology is sound. This seems vastly more reasonable, though obviously it's only possible if you know exactly what you want to do in advance. So why not switch toward reviewing work earlier, through registered reports or collaboration, and end the artificial bottleneck at publication? It would prevent wasted time on flawed experiments and remove some of the bias against negative results. (Of course, the other problem is that peer reviewers aren't very good at detecting errors -- even serious ones. Medicine has moved towards standard checklist for common types of research, requiring papers to report every important methodological detail so review can be more thorough. But I don't think you'll improve the overall quality of peer review without changing the current "fine, I'll review it in between writing three grant proposals and grading 150 exams" voluntary review culture.

Kalium 11y ago
When collaborating, people become attached to their work. This inevitably clouds their judgment when it comes time to review, especially given that financial ties are often involved. External peer review reduces those problems.
semi-extrinsic 11y ago
Wow, that Registered Reports thing sounds like a fabulous idea. I think it could be an interesting solution to the problem that it's really hard to publish things that don't work. Several times I've read papers thinking "hmm, this might be worth trying", and then talking to the authors you hear "no, no, it's a crappy method, try that other thing instead".

Kalium 11y ago

When collaborating, people become attached to their work. This inevitably clouds their judgment when it comes time to review, especially given that financial ties are often involved. External peer review reduces those problems.

semi-extrinsic 11y ago

Wow, that Registered Reports thing sounds like a fabulous idea. I think it could be an interesting solution to the problem that it's really hard to publish things that don't work. Several times I've read papers thinking "hmm, this might be worth trying", and then talking to the authors you hear "no, no, it's a crappy method, try that other thing instead".