3 ms·

If reviewers are de-anonymized, the opposite of what you think will happen. Nobody other than the most socially stunted would be willing to write a harsh review

If reviewers are de-anonymized, the opposite of what you think will happen. Nobody other than the most socially stunted would be willing to write a harsh review and make an enemy for life. People are petty. Most fields in academia are incredibly small, with everyone knowing each other. Reviews would become way more positive and useless. Collusion for positive review is pretty rare and wouldn’t really change with real names anyway because it’s pretty subjective to claim a positive judgement is obviously the result of collusion. It’s also already the job of the editors to collate the reviews and make his own judgment based on them. If one review is overly positive and adds no useful information, it can and will be discarded. The editor can even call for another reviewer if he/she is still undecided. I’ve reviewed papers and recommended acceptance, yet the paper was still rejected; the opposite has happened too. As a reviewer you are only advising the editor.

Beldin 5y ago
> Reviews would become way more positive and useless. In my corner of CS, there are plenty reviews of debatable use that are very negative. In fact, I have a pet hypothesis that the average score of the lower scoring but accepted papers is negative (scale from -3 to 3). And I wouldn't be surprised if the median paper's score was negative. Positive uselessness doesn't seem like that much worse.
dahart 5y ago
Are you sure about that? I’m not sure you are hearing and understanding my suggestion. A review would always consist of one paper being ranked above another one. The tone of the review comments would have no bearing on the usefulness of the review. The reason we even have critiques in reviews is to justify the score that the reviewer assigns. Reviewers are specifically asked to produce reasons. What I’m suggesting is different because reviewers would rank two papers against each other and not be expected to justify their decision, nor would they be asked to provide a magnitude for their opinion. Comments and critiques would be allowed, but there’s no need for harsh reviews. I’ve seen a lot of overly and unnecessarily harsh reviews. Anonymizing enables over-stating criticism, it happens routinely. I don’t think I agree that harsh critique is necessary for a healthy review system. It is already the case that good reviews are not extremely harsh, they focus on the facts and are willing to stand by their statements. I don’t personally know that many researchers who have trouble being direct in person and face to face, or of offering constructive criticism. > As a reviewer you are only advising the editor This completely depends on the journal or conference. Quite a few of them, especially the larger ones, do not override reviews casually nor often. And what I’m suggesting is a system where this idea can change, where editors can more easily trust the review results, and won’t need to override the decision. > People are petty. This might well be true. And so I’m not entirely understanding your argument. It seems to be simultaneously suggesting we have a problem, and defending the status quo as the way it needs to be. What would you suggest as a way to improve the review system so that pettiness has less influence than it does today?
- dasudasu 5y ago
  Ranking only makes sense in a venue where the number of papers or presentations is limited. There many online journals of good reputation that will publish anything over a certain (obviously not perfectly well-defined) bar. The number of papers tends to increase over the years as the numbers of researchers also increase, and this is considered acceptable. Other journals are so selective that they won’t publish anything if nothing is deemed worthy. You’re also misunderstanding the point of reviews. They also serve as comments to the authors to modify their manuscript and make it acceptable for publication.

Beldin 5y ago

> Reviews would become way more positive and useless. In my corner of CS, there are plenty reviews of debatable use that are very negative. In fact, I have a pet hypothesis that the average score of the lower scoring but accepted papers is negative (scale from -3 to 3). And I wouldn't be surprised if the median paper's score was negative. Positive uselessness doesn't seem like that much worse.

dahart 5y ago

Are you sure about that? I’m not sure you are hearing and understanding my suggestion. A review would always consist of one paper being ranked above another one. The tone of the review comments would have no bearing on the usefulness of the review. The reason we even have critiques in reviews is to justify the score that the reviewer assigns. Reviewers are specifically asked to produce reasons. What I’m suggesting is different because reviewers would rank two papers against each other and not be expected to justify their decision, nor would they be asked to provide a magnitude for their opinion. Comments and critiques would be allowed, but there’s no need for harsh reviews. I’ve seen a lot of overly and unnecessarily harsh reviews. Anonymizing enables over-stating criticism, it happens routinely. I don’t think I agree that harsh critique is necessary for a healthy review system. It is already the case that good reviews are not extremely harsh, they focus on the facts and are willing to stand by their statements. I don’t personally know that many researchers who have trouble being direct in person and face to face, or of offering constructive criticism. > As a reviewer you are only advising the editor This completely depends on the journal or conference. Quite a few of them, especially the larger ones, do not override reviews casually nor often. And what I’m suggesting is a system where this idea can change, where editors can more easily trust the review results, and won’t need to override the decision. > People are petty. This might well be true. And so I’m not entirely understanding your argument. It seems to be simultaneously suggesting we have a problem, and defending the status quo as the way it needs to be. What would you suggest as a way to improve the review system so that pettiness has less influence than it does today?

dasudasu 5y ago
Ranking only makes sense in a venue where the number of papers or presentations is limited. There many online journals of good reputation that will publish anything over a certain (obviously not perfectly well-defined) bar. The number of papers tends to increase over the years as the numbers of researchers also increase, and this is considered acceptable. Other journals are so selective that they won’t publish anything if nothing is deemed worthy. You’re also misunderstanding the point of reviews. They also serve as comments to the authors to modify their manuscript and make it acceptable for publication.