The described result is not in conflict with decisions being reasonably consistent. Suppose that two reviewers independently rank papers 80% on quality and 20%

The described result is not in conflict with decisions being reasonably consistent. Suppose that two reviewers independently rank papers 80% on quality and 20% on chance factors. With good odds, the two reviewers will agree with each other on the relative rankings of any given pair of papers. But their lists of the top 10% of papers will largely not be in agreement with each other.