Thanks for the comment! I looked at WHOOPPEE a while ago and it is great work. We have thought a lot about doing rankings instead of grading/assessing each piec

Thanks for the comment! I looked at WHOOPPEE a while ago and it is great work. We have thought a lot about doing rankings instead of grading/assessing each piece of work individually with evaluation criteria. On one hand, I believe that it is easier with rankings to get more numerically correct grades. Humans are bad at rating things, but binary rankings are doable. The challenge with rankings for peer review, is two-fold. The first problem is that a ranking does not directly tell you where you are lacking or how to improve. The second problem is that quality is generally not 1-dimensional (as the ranking would be). These problems can be solved by letting students rank papers according to multiple criteria (consequently generating multiple rankings) and then combining these rankings. The challenge with this solution is that doing multiple rankings often ends up becoming a quite time-consuming and overwhelming task. Also, from the research I have looked at about this, students generally find it hard to pick one or the other, often requesting the "I don't really know" option making the problem somewhat less trivial.