3 ms·
I don't like Elsevier, but I always wondered: Why aren't there a few meta-reviewers, to review the reviews? That is, why doesn't the PC chair, or some key rev
by einpoklum 7y ago
I don't like Elsevier, but I always wondered:
Why aren't there a few meta-reviewers, to review the reviews?
That is, why doesn't the PC chair, or some key reviewers, have a look at the reviews given on papers to make sure that:
- They were respectful
- They were to the point
- They did not indicate obvious lack of knowledge of the relevant subject matter (this one might be a bit tricky)
- They were not self-promoting
That would really have helped a lot of conferences...
(but I actually know the answer: There's a dearth of reviewers as-is, so there's de-motivation to introduce this meta-review mechanism even if it would help review quality.)
- Majromax 7y ago> Why aren't there a few meta-reviewers, to review the reviews? In principle, the associate editor managing the submission (and making the ultimate accept/reject decision) should be doing this for every paper. In practice, the workload is such that editors have an incentive to make the review process as automatic as possible on their end.
- einpoklum 7y agoSo in theory, a highly-trusted reviewer should avoid regular reviews and be engaged in meta-reviews, because of that load.
- lhorie 7y agoIf the problem relates to the integrity of reviewers, I suspect you can't fundamentally fix the problem by adding more reviewers.
- einpoklum 7y agoSure you can: People examining each other's work deter corruption and also have a good chance of noticing it. Only if a reviewer and a meta-reviewer collude to mis-review do you have a problem, and that should be rather rare.
- lhorie 7y agoSure, in theory. Another theoretical but entirely possible situation is you get a black market of colluding citations. Consider this analogy: a company has workers and it wants to ensure they are working efficiently instead of chasing promotions. So they add managers. And managers for the managers. And manager for managers for managers. But in the end, everything is drowning in inefficiencies because everyone is still chasing promotions. Without addressing the root problem, it won't help to add more of the type of individuals who are prone to succumb to perverse incentives.