3 ms·
That would only work if the review decisions could be expected to be reasonably consistent from person to person. But, from the article: > In a well-publicized
by snowwrestler 5y ago
That would only work if the review decisions could be expected to be reasonably consistent from person to person. But, from the article:
> In a well-publicized case in 2014, organizers of the Neural Information Processing Systems Conference formed two independent program committees and had 10% of submissions reviewed by both. The result was that almost 60% of papers accepted by one program committee were rejected by the other, suggesting that the fate of many papers is determined by the specifics of the reviewers selected and not just the inherent value of the work itself.
With this much demonstrated discrepancy between two sets of reviewers, it’s hard to believe that adding a validation step would produce a consistent improvement. How can people be expected to find improperly accepted papers when they have less than 50% agreement on the acceptance of good-faith submissions?
Honestly I think this seeming randomness in acceptance is at the heart of why people might think cheating is acceptable. If the process is not reliable, why bother submitting to it?
- sjg007 5y ago>The result was that almost 60% of papers accepted by one program committee were rejected by the other, suggesting that the fate of many papers is determined by the specifics of the reviewers selected and not just the inherent value of the work itself. I referenced Kahneham's latest book, Noise, above but this is exactly the problem he focuses on. There are solutions.
- cortesoft 5y agoWouldn’t a ‘solution’ only work if there actually is some underlying objective quality to a paper? It might be the case that different reviewers disagree because there is no right answer, because there is no such thing as objective quality.
- sjg007 5y agoWell wisdom of the crowds suggest that there is a truth... and a conference is charged with representing the best so I think that randomization and meta moderation + other strategies are better in the long run.
- btilly 5y agoThe described result is not in conflict with decisions being reasonably consistent. Suppose that two reviewers independently rank papers 80% on quality and 20% on chance factors. With good odds, the two reviewers will agree with each other on the relative rankings of any given pair of papers. But their lists of the top 10% of papers will largely not be in agreement with each other.
- visarga 5y agoThis is in aggregate. If you look at great papers, they are almost surely accepted. Bad papers are almost surely rejected. But those in the middle are a coin toss.