3 ms·

But who decides what's constructive? It's all very well if you're choosing amongst people who basically agree with you - what if they're telling you it's going

But who decides what's constructive? It's all very well if you're choosing amongst people who basically agree with you - what if they're telling you it's going to take another year of experiments to get it up to standard? Which reviewers, too, do you respond to? If it's a popular paper you probably can't respond to all criticism, but picking which criticism to interact with isn't going to lead to good science. There's a lot of value, even if it's not perfect, in writers being forced to respond to one or two reviewers' comments.

danieltillett 9y ago
I suggested in the post either the editor or the other reviewers makes the decision of which was the most constructive review. The suggestion of going back and doing more work exists under the current peer review system. With my proposal you might even get suggestions of what work needs to be done and why you should do it. In all the papers I have published I have only had a few reviewers agree on anything. The normal approach is you agree to all the reasonable demands even if you don’t personally like them and argue over the important one. Most of the time if the reviewers see you have moved some of the way towards their arbitrary demands they will let you hold onto the important ones.
- wyattpeak 9y ago
  You're right, I completely missed that. I quite like the idea of other reviewers picking which is best, although it may take some fiddling to avoid abuse. I wasn't suggesting reviewers currently can't tell you to go back and do more work, quite the opposite - that I think it's a valuable if often too arbitrary requirement. But if the writer isn't picking which comments to respond to my point is moot anyway.
tom_mellior 9y ago
> But who decides what's constructive? For a start, it would be good if as an author I could give feedback on review quality. Rebuttal periods are becoming standard for computer science conferences, and you can put some feedback into your rebuttal, but if you are too frank about bad reviewers, they could decide to recommend rejection. It would be good to add an option to review reviewers after the final decision on the paper has been made. I do think that that would make (some) reviewers more constructive. A few conferences do have distinguished reviewer awards, though I've never really seen explained how the winners are chosen.

danieltillett 9y ago

I suggested in the post either the editor or the other reviewers makes the decision of which was the most constructive review. The suggestion of going back and doing more work exists under the current peer review system. With my proposal you might even get suggestions of what work needs to be done and why you should do it. In all the papers I have published I have only had a few reviewers agree on anything. The normal approach is you agree to all the reasonable demands even if you don’t personally like them and argue over the important one. Most of the time if the reviewers see you have moved some of the way towards their arbitrary demands they will let you hold onto the important ones.

wyattpeak 9y ago
You're right, I completely missed that. I quite like the idea of other reviewers picking which is best, although it may take some fiddling to avoid abuse. I wasn't suggesting reviewers currently can't tell you to go back and do more work, quite the opposite - that I think it's a valuable if often too arbitrary requirement. But if the writer isn't picking which comments to respond to my point is moot anyway.

tom_mellior 9y ago

> But who decides what's constructive? For a start, it would be good if as an author I could give feedback on review quality. Rebuttal periods are becoming standard for computer science conferences, and you can put some feedback into your rebuttal, but if you are too frank about bad reviewers, they could decide to recommend rejection. It would be good to add an option to review reviewers after the final decision on the paper has been made. I do think that that would make (some) reviewers more constructive. A few conferences do have distinguished reviewer awards, though I've never really seen explained how the winners are chosen.