3 ms·
Peer reviews seem to be very hit and miss. I guess it's like asking why a software bug wasn't found during code review, except code reviews are somewhat strong
by mike_hearn 6y ago
Peer reviews seem to be very hit and miss. I guess it's like asking why a software bug wasn't found during code review, except code reviews are somewhat stronger in the sense that everyone involved wants the final result to actually function.
A clear problem I've seen when meta-reviewing peer reviews is that the goal seems to be demonstrating collegial constructivity, in the sense that things that are obviously totally wrong and yield a useless paper will be highlighted as sub-optimal or skipped over, in favour of smaller things that are fixable. That's understandable from the perspective of keeping friends and building influence with peers, all of which are critical to academic success, but it's pretty fatal for wider trust in science.
Another problem is that peer reviews are extremely slow. Several months on average. If someone takes 3 months to review a paper, and then addressing the comments takes another month, there is a lot of pressure in the system to go straight to publication at that point - in a strict code review environment the reviewer might observe that some of the problems still weren't fixed and send bad work back four or five times, often easily more. But that's because code review has a fast turnaround time. You can't do that with the speed of scientific peer reviews.
On the plus side the reviews do tend to be pretty comprehensive and seem to be taken seriously within their own standards. Part of why they take so long is they're often pretty long. The issue isn't scientists half-assing it, the problems are deeper and more psychological.
- tpoacher 6y ago> if someone takes 3 months to review a paper I have the exact opposite experience/ problem. Whenever I'm asked to review, there s an artificially short deadline involved (typically a couple of weeks). Which, assuming you actually work for a living, effectively equates to having a couple of days at the weekend to review a paper.
- mike_hearn 6y agoHmm, interesting. My number comes from a meta-analysis of peer reviews that compared usual latency to COVID-paper latency. However they might have been looking at the delay between the paper being finalised and the review being committed, or something like that. There's presumably a lot of latency added by the journals finding and routing papers to reviewers.