4 ms·

Yes, when I looked into it I found there is actually zero evidence that institutionalized peer review is helpful at all. In fact, it looked like it's primary ro

by nonbel 9y ago

Yes, when I looked into it I found there is actually zero evidence that institutionalized peer review is helpful at all. In fact, it looked like it's primary role was to impede science by enforcing whatever prevailing bias exists at the moment. For example: >"In addition to being poor at detecting gross defects and almost useless for detecting fraud it is slow, expensive, profligate of academic time, highly subjective, something of a lottery, prone to bias, and easily abused." https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1420798/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1420798/

BeetleB 9y ago
>Yes, when I looked into it I found there is actually zero evidence that institutionalized peer review is helpful at all. In fact, it looked like it's primary role was to impede science by enforcing whatever prevailing bias exists at the moment. While I've experienced what you've described, I think throwing it all out is merely going to the other extreme. I think peer review should remain, but may need reform. The larger problem is the attitude of scientists today - some of the problems you yourself mentioned. Removing peer review will not fix those. And "fixing" peer review to handle those problems is merely treating a symptom. Ultimately, the larger problem is the mentality that has spread amongst academics - and the problems will remain no matter how you tweak peer review. >In addition to being poor at detecting gross defects and almost useless for detecting fraud This gets mentioned a lot, so I'll repeat what many others have said: Peer review was never meant to detect fraud, and should not be expected to. Back during the Schon scandal, people kept asking how peer review had failed the system to let him get so far before he was caught. And the answer was always the same: Peer review is not the appropriate method to catch such people. It was never meant for it. https://en.wikipedia.org/wiki/Sch%C3%B6n_scandal https://en.wikipedia.org/wiki/Sch%C3%B6n_scandal
- nonbel 9y ago
  I agree that peer review should not be used to detect fraud. I would even say there is no reason to have any method devoted to detecting fraud. Instead just use independent replication to filter out fraudulent, sloppy, and unreliable results.
  - BeetleB 9y ago
    The problem with that is debunked papers continue to be heavily cited.
    - nonbel 9y ago
      If it were usual to do the independent replication, I don't see how it could be justified to cite one but not the other if the results conflicted.
JohnStrange 9y ago
I've met many critiques of the peer-review system (including myself), but have never heard a compelling suggestion with what to replace it and why the replacement would be better. Without peer review, editors would have to assign papers by gut feeling or perceived reputation of the author, or the scientific community would be flooded with massive amounts of papers that are way below acceptable quality or even entirely false, not the speak of the dangers of pseudo-science and fake scientists or even paid shills. However, I believe open access journals can and should experiment with alternative forms of peer review. For example, forcing every submitter to anonymously score 5 papers before his/her own submission is accepted, a karma system, a good meta-moderation system, and a good document revision system taken together could work for online publications. There is a lot of potential for innovation in online publishing. The problems are in the end similar to those on online forums and social media. It's hard to keep trolls and people who try to game the system away and equally hard to prevent people from scoring solely according to their personal tastes and with kind of 'political' (as in politics of science) motivations, especially when the process is anonymous. But ordinary peer review has the same problem.
- kxyvr 9y ago
  I believe that there are simple modifications that could be done now to the existing system. Namely, why are reviews typically single blind, where the author doesn't know the referees, but not double blind? Certainly, the reviewers could figure out who wrote the paper based on the context, but it seems inexcusable that we don't even attempt to remove this bias.
- nonbel 9y ago
  >"have never heard a compelling suggestion with what to replace it and why the replacement would be better" You can't just replace/remove peer review and expect things to improve. It is part of the whole "assembly line" science environment that was created when the US government became the primary source of funding in the 1940s or so. If we want functioning science, we need to replace at least two things: 1) The search for "significant" p-values with comparing precise predictions to data. 2) Peer review with independent replication. Both are pretty much just reverting to how things worked earlier.

by nonbel 9y ago

BeetleB 9y ago

>Yes, when I looked into it I found there is actually zero evidence that institutionalized peer review is helpful at all. In fact, it looked like it's primary role was to impede science by enforcing whatever prevailing bias exists at the moment. While I've experienced what you've described, I think throwing it all out is merely going to the other extreme. I think peer review should remain, but may need reform. The larger problem is the attitude of scientists today - some of the problems you yourself mentioned. Removing peer review will not fix those. And "fixing" peer review to handle those problems is merely treating a symptom. Ultimately, the larger problem is the mentality that has spread amongst academics - and the problems will remain no matter how you tweak peer review. >In addition to being poor at detecting gross defects and almost useless for detecting fraud This gets mentioned a lot, so I'll repeat what many others have said: Peer review was never meant to detect fraud, and should not be expected to. Back during the Schon scandal, people kept asking how peer review had failed the system to let him get so far before he was caught. And the answer was always the same: Peer review is not the appropriate method to catch such people. It was never meant for it. https://en.wikipedia.org/wiki/Sch%C3%B6n_scandal https://en.wikipedia.org/wiki/Sch%C3%B6n_scandal

nonbel 9y ago
I agree that peer review should not be used to detect fraud. I would even say there is no reason to have any method devoted to detecting fraud. Instead just use independent replication to filter out fraudulent, sloppy, and unreliable results.
- BeetleB 9y ago
  The problem with that is debunked papers continue to be heavily cited.
  - nonbel 9y ago
    If it were usual to do the independent replication, I don't see how it could be justified to cite one but not the other if the results conflicted.

JohnStrange 9y ago

I've met many critiques of the peer-review system (including myself), but have never heard a compelling suggestion with what to replace it and why the replacement would be better. Without peer review, editors would have to assign papers by gut feeling or perceived reputation of the author, or the scientific community would be flooded with massive amounts of papers that are way below acceptable quality or even entirely false, not the speak of the dangers of pseudo-science and fake scientists or even paid shills. However, I believe open access journals can and should experiment with alternative forms of peer review. For example, forcing every submitter to anonymously score 5 papers before his/her own submission is accepted, a karma system, a good meta-moderation system, and a good document revision system taken together could work for online publications. There is a lot of potential for innovation in online publishing. The problems are in the end similar to those on online forums and social media. It's hard to keep trolls and people who try to game the system away and equally hard to prevent people from scoring solely according to their personal tastes and with kind of 'political' (as in politics of science) motivations, especially when the process is anonymous. But ordinary peer review has the same problem.

kxyvr 9y ago
I believe that there are simple modifications that could be done now to the existing system. Namely, why are reviews typically single blind, where the author doesn't know the referees, but not double blind? Certainly, the reviewers could figure out who wrote the paper based on the context, but it seems inexcusable that we don't even attempt to remove this bias.
nonbel 9y ago
>"have never heard a compelling suggestion with what to replace it and why the replacement would be better" You can't just replace/remove peer review and expect things to improve. It is part of the whole "assembly line" science environment that was created when the US government became the primary source of funding in the 1940s or so. If we want functioning science, we need to replace at least two things: 1) The search for "significant" p-values with comparing precise predictions to data. 2) Peer review with independent replication. Both are pretty much just reverting to how things worked earlier.