3 ms·

There are a lot of good arguments [0] that peer review doesn't work, and leads to worse outcomes than just publishing openly and letting the marketplace of idea

by dimal 3y ago

There are a lot of good arguments [0] that peer review doesn't work, and leads to worse outcomes than just publishing openly and letting the marketplace of ideas decide which papers are actually worthwhile. Think about it. A small group of gatekeepers decides what research is worthwhile and what is not. How well does that kind of gatekeeping work in other areas? For example, Einstein only had one paper peer reviewed (which was rejected) and things turned out well for him. What if his papers had gone through a committee? [0] https://www.experimental-history.com/p/the-rise-and-fall-of-peer-review https://www.experimental-history.com/p/the-rise-and-fall-of-...

jltsiren 3y ago
The group of gatekeepers is not small. When Science Twitter was still a thing, two common topics were editors complaining about how difficult it is to find peer reviewers and academics complaining how many review requests they get. Pretty much every established researcher is involved in the gatekeeping, and they can devote as much time to it as they want. In many fields, preprints have been an established practice for a long time. That allows us to see the alternative to peer review. The main variables that predict how much attention your preprint gets are your name and the topic. With all its imperfections, peer review at least gives a second chance for less known researchers working on less fashionable topics.
bluefirebrand 3y ago
There's also an argument to be made that Peer Review is absolutely failing, because of the replication crisis https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis
matthewdgreen 3y ago
> There are a lot of good arguments [0] that peer review doesn't work That is not a good argument. It points out that research productivity has changed and then tries to correlate this with the rise of peer review, before admitting that the entire argument is confounded and pointless. Then it goes on to the standard non-practitioner approach and cherry-picks examples where peer review didn’t find flaws, which I mean, duh. Of course peer review doesn’t work perfectly to find all errors and fraud, nobody is arguing the point. You could write a similar post demonstrating that fire departments don’t fight fires, or that restaurant health codes aren’t needed to prevent food poisoning.

by dimal 3y ago

jltsiren 3y ago

The group of gatekeepers is not small. When Science Twitter was still a thing, two common topics were editors complaining about how difficult it is to find peer reviewers and academics complaining how many review requests they get. Pretty much every established researcher is involved in the gatekeeping, and they can devote as much time to it as they want. In many fields, preprints have been an established practice for a long time. That allows us to see the alternative to peer review. The main variables that predict how much attention your preprint gets are your name and the topic. With all its imperfections, peer review at least gives a second chance for less known researchers working on less fashionable topics.

bluefirebrand 3y ago

There's also an argument to be made that Peer Review is absolutely failing, because of the replication crisis https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis

matthewdgreen 3y ago

> There are a lot of good arguments [0] that peer review doesn't work That is not a good argument. It points out that research productivity has changed and then tries to correlate this with the rise of peer review, before admitting that the entire argument is confounded and pointless. Then it goes on to the standard non-practitioner approach and cherry-picks examples where peer review didn’t find flaws, which I mean, duh. Of course peer review doesn’t work perfectly to find all errors and fraud, nobody is arguing the point. You could write a similar post demonstrating that fire departments don’t fight fires, or that restaurant health codes aren’t needed to prevent food poisoning.