Indeed, I've seen at least one paper where they used Mecanical Turk and aggregated results (by showing the same examples to multiple people) to check for qualit

Indeed, I've seen at least one paper where they used Mecanical Turk and aggregated results (by showing the same examples to multiple people) to check for quality. It sounds like they did almost exactly the same thing here. Paper: 'Antisocial Behaviour In Online Discussion Communities' (Cheng et al., 2015) Link: http://arxiv.org/pdf/1504.00680v1.pdf http://arxiv.org/pdf/1504.00680v1.pdf (Search for 'Mechanical Turk' / section Data Preparation -> Measuring text quality.)