4 ms·

The big problem with peer review is almost no reviewing takes place and it is at best is either a way for your enemies to get revenge or a cheap proofreading se

The big problem with peer review is almost no reviewing takes place and it is at best is either a way for your enemies to get revenge or a cheap proofreading service. The solution is to offer a largish reward to the most constructive peer review for each manuscript chosen by the editor, or better by other reviewers. If I (or more likely one of my starving grad students) knew that I would get $1000 for providing the most constructive peer review then I would put in the required effort to make peer review a worthwhile exercise. One thing that would help the peer review process is a data dump of the raw data. Many times I have seen some figure or table in a manuscript and felt I could not provide a constructive critic without seeing the underlying data. Going back through the editor to get this data is just too much of a hassle most of the time.

wyattpeak 9y ago
But who decides what's constructive? It's all very well if you're choosing amongst people who basically agree with you - what if they're telling you it's going to take another year of experiments to get it up to standard? Which reviewers, too, do you respond to? If it's a popular paper you probably can't respond to all criticism, but picking which criticism to interact with isn't going to lead to good science. There's a lot of value, even if it's not perfect, in writers being forced to respond to one or two reviewers' comments.
- danieltillett 9y ago
  I suggested in the post either the editor or the other reviewers makes the decision of which was the most constructive review. The suggestion of going back and doing more work exists under the current peer review system. With my proposal you might even get suggestions of what work needs to be done and why you should do it. In all the papers I have published I have only had a few reviewers agree on anything. The normal approach is you agree to all the reasonable demands even if you don’t personally like them and argue over the important one. Most of the time if the reviewers see you have moved some of the way towards their arbitrary demands they will let you hold onto the important ones.
  - wyattpeak 9y ago
    You're right, I completely missed that. I quite like the idea of other reviewers picking which is best, although it may take some fiddling to avoid abuse. I wasn't suggesting reviewers currently can't tell you to go back and do more work, quite the opposite - that I think it's a valuable if often too arbitrary requirement. But if the writer isn't picking which comments to respond to my point is moot anyway.
- tom_mellior 9y ago
  > But who decides what's constructive? For a start, it would be good if as an author I could give feedback on review quality. Rebuttal periods are becoming standard for computer science conferences, and you can put some feedback into your rebuttal, but if you are too frank about bad reviewers, they could decide to recommend rejection. It would be good to add an option to review reviewers after the final decision on the paper has been made. I do think that that would make (some) reviewers more constructive. A few conferences do have distinguished reviewer awards, though I've never really seen explained how the winners are chosen.

wyattpeak 9y ago

But who decides what's constructive? It's all very well if you're choosing amongst people who basically agree with you - what if they're telling you it's going to take another year of experiments to get it up to standard? Which reviewers, too, do you respond to? If it's a popular paper you probably can't respond to all criticism, but picking which criticism to interact with isn't going to lead to good science. There's a lot of value, even if it's not perfect, in writers being forced to respond to one or two reviewers' comments.

danieltillett 9y ago
I suggested in the post either the editor or the other reviewers makes the decision of which was the most constructive review. The suggestion of going back and doing more work exists under the current peer review system. With my proposal you might even get suggestions of what work needs to be done and why you should do it. In all the papers I have published I have only had a few reviewers agree on anything. The normal approach is you agree to all the reasonable demands even if you don’t personally like them and argue over the important one. Most of the time if the reviewers see you have moved some of the way towards their arbitrary demands they will let you hold onto the important ones.
- wyattpeak 9y ago
  You're right, I completely missed that. I quite like the idea of other reviewers picking which is best, although it may take some fiddling to avoid abuse. I wasn't suggesting reviewers currently can't tell you to go back and do more work, quite the opposite - that I think it's a valuable if often too arbitrary requirement. But if the writer isn't picking which comments to respond to my point is moot anyway.
tom_mellior 9y ago
> But who decides what's constructive? For a start, it would be good if as an author I could give feedback on review quality. Rebuttal periods are becoming standard for computer science conferences, and you can put some feedback into your rebuttal, but if you are too frank about bad reviewers, they could decide to recommend rejection. It would be good to add an option to review reviewers after the final decision on the paper has been made. I do think that that would make (some) reviewers more constructive. A few conferences do have distinguished reviewer awards, though I've never really seen explained how the winners are chosen.