3 ms·
I'm a recent PhD graduate that focused heavily on publication and collected ~20 publications. I think the situation with peer review is worse than discussed he
by tomlue 7y ago
I'm a recent PhD graduate that focused heavily on publication and collected ~20 publications. I think the situation with peer review is worse than discussed here.
Peer review:
1. dramatically reduces the pace of progress
2. exacerbates publication bias
3. creates a false sense of accomplishment
4. creates a false meritocracy
5. creates many perverse incentives
(3) and (4) unfortunately wrap into grant financing as well. The merit of your research isn't measured on its impact on the human condition, but on it's 'impact' factor (a measure derived from your publications).
The gatekeeper problem here is pernicious as well. If you become a highly cited author your ability to get/maintain financing improves. You also become a gatekeeper as a peer reviewer. Which means that you are now strongly incentivized to accept papers from people who cite your work or align themselves with you, and reject everything else.
What is absolutely amazing here, is that the peer review process is opaque. It is my belief that if you knew who reviewed which papers you would quickly discover that mild to severe abuse of peer review is the norm, not action by a handful of bad actors. This is because the entire academic reward system is wrapped into the process. Getting your name on a big paper can have lifelong ramifications on your ability to get grants, start companies, do consulting work, etc.
Peer reviewers should probably get paid for the work. If they don't get paid then their incentive to do review must come from somewhere else, vague notions of improving the field don't cut it. Peer review should obviously be transparent. Some people might be uncomfortable signing their name to a paper rejection, but its time to get over that. A small payment might help reviewers overcome this discomfort. It is bizarre that peer reviewers don't get paid. Peer review is valuable work if done right, and without payment all the reward of being a peer reviewer comes for the wrong reasons.
Finally, I agree with some other comments. Publication should not be contingent on peer review, it should come first. This would increase the pace of progress, reduce publication bias, reduce the false meritocracy, reduce the ability for bad actors to censor research, and more. The cost would be a larger number of publications, but perhaps this would help people realize that many of the publications coming out right now are of little value.
Transparency is badly needed in many facets of academic research. My company made a site that helps bring transparency into literature review (sysrev.com).
- sgillen 7y agoI totally agree with the transparency thing. I would love to see something like open review (https://openreview.net/forum?id=ryQu7f-RZ https://openreview.net/forum?id=ryQu7f-RZ) become more common, and even have something like this replace the traditional peer review system. We would all have to think long and hard about how to align incentives to get the desired results.As a first pass I think hiring some amount of people to review every paper, and then allowing anyone who wants to come by and leave their comments as well might be a place to start. I think some sort of reputation system similar to stack overflow might be a motivator as well.
- Fomite 7y agoI've badly soured on open peer review, having had some experiences with it. I think you'll have very differential levels of review, and very differential values of how free people feel to go after authors. I'd much prefer double-blind peer review.
- cassowary37 7y agoThose are all very real problems (and as you point out, they impact grant review as well). On the other hand, fully transparent peer review doesn't necessary address most of them. Reviewers who have to sign their reviews may be reluctant to anger colleagues (or try to curry favor with them). As you presumably know a many journals have experimented with open peer review, but editors still need to police the reviews to look for bias. It solves some problems but creates others.
- tomlue 7y agotransparent review isn't a perfect solution. But the problem you suggest of reluctance to anger colleagues or currying favor are worse under the current system than they would be under a transparent one. Just because other journals failed in the past doesn't mean we shouldn't try again in the future! Maybe there were some mistakes we could learn from? Also the internet is becoming a more familiar tool used more often by more people every day, maybe it just wasn't the right time previously.