3 ms·

I've met many critiques of the peer-review system (including myself), but have never heard a compelling suggestion with what to replace it and why the replaceme

I've met many critiques of the peer-review system (including myself), but have never heard a compelling suggestion with what to replace it and why the replacement would be better. Without peer review, editors would have to assign papers by gut feeling or perceived reputation of the author, or the scientific community would be flooded with massive amounts of papers that are way below acceptable quality or even entirely false, not the speak of the dangers of pseudo-science and fake scientists or even paid shills. However, I believe open access journals can and should experiment with alternative forms of peer review. For example, forcing every submitter to anonymously score 5 papers before his/her own submission is accepted, a karma system, a good meta-moderation system, and a good document revision system taken together could work for online publications. There is a lot of potential for innovation in online publishing. The problems are in the end similar to those on online forums and social media. It's hard to keep trolls and people who try to game the system away and equally hard to prevent people from scoring solely according to their personal tastes and with kind of 'political' (as in politics of science) motivations, especially when the process is anonymous. But ordinary peer review has the same problem.

kxyvr 9y ago
I believe that there are simple modifications that could be done now to the existing system. Namely, why are reviews typically single blind, where the author doesn't know the referees, but not double blind? Certainly, the reviewers could figure out who wrote the paper based on the context, but it seems inexcusable that we don't even attempt to remove this bias.
nonbel 9y ago
>"have never heard a compelling suggestion with what to replace it and why the replacement would be better" You can't just replace/remove peer review and expect things to improve. It is part of the whole "assembly line" science environment that was created when the US government became the primary source of funding in the 1940s or so. If we want functioning science, we need to replace at least two things: 1) The search for "significant" p-values with comparing precise predictions to data. 2) Peer review with independent replication. Both are pretty much just reverting to how things worked earlier.
- BeetleB 9y ago
  >The search for "significant" p-values with comparing precise predictions to data. Problems: 1. Not sure we have a decent alternative. 2. While p-values are heavily used in some fields (social sciences, medicine, etc) they are not in others (e.g. physics). Yet those other disciplines still have similar problems.
  - nonbel 9y ago
    What do you mean about not having a decent alternative? The perfect alternative is given in the second part of what you quoted. Also, I see that physics is now starting to rely on statistical significance (eg LIGO), which I guarantee will be to their detriment. They are just behind those other fields in adopting it.
BeetleB 9y ago
>or the scientific community would be flooded with massive amounts of papers that are way below acceptable quality or even entirely false, not the speak of the dangers of pseudo-science and fake scientists or even paid shills. All problems that exist even with peer review. >For example, forcing every submitter to anonymously score 5 papers before his/her own submission is accepted This leads to low quality reviews. >a karma system, a good meta-moderation system, On the whole Internet, I have yet to see a system of karma/moderation that lasts. They are always gamed in the end. And I'm not even sure what you mean here. What will high karma provide to the researcher?

kxyvr 9y ago

I believe that there are simple modifications that could be done now to the existing system. Namely, why are reviews typically single blind, where the author doesn't know the referees, but not double blind? Certainly, the reviewers could figure out who wrote the paper based on the context, but it seems inexcusable that we don't even attempt to remove this bias.

nonbel 9y ago

>"have never heard a compelling suggestion with what to replace it and why the replacement would be better" You can't just replace/remove peer review and expect things to improve. It is part of the whole "assembly line" science environment that was created when the US government became the primary source of funding in the 1940s or so. If we want functioning science, we need to replace at least two things: 1) The search for "significant" p-values with comparing precise predictions to data. 2) Peer review with independent replication. Both are pretty much just reverting to how things worked earlier.

BeetleB 9y ago
>The search for "significant" p-values with comparing precise predictions to data. Problems: 1. Not sure we have a decent alternative. 2. While p-values are heavily used in some fields (social sciences, medicine, etc) they are not in others (e.g. physics). Yet those other disciplines still have similar problems.
- nonbel 9y ago
  What do you mean about not having a decent alternative? The perfect alternative is given in the second part of what you quoted. Also, I see that physics is now starting to rely on statistical significance (eg LIGO), which I guarantee will be to their detriment. They are just behind those other fields in adopting it.

BeetleB 9y ago

>or the scientific community would be flooded with massive amounts of papers that are way below acceptable quality or even entirely false, not the speak of the dangers of pseudo-science and fake scientists or even paid shills. All problems that exist even with peer review. >For example, forcing every submitter to anonymously score 5 papers before his/her own submission is accepted This leads to low quality reviews. >a karma system, a good meta-moderation system, On the whole Internet, I have yet to see a system of karma/moderation that lasts. They are always gamed in the end. And I'm not even sure what you mean here. What will high karma provide to the researcher?