We used some of the research around Mechanical Turk to find best practices for limiting trolling (e.g. [1]). Our approach thus far has been the two-thirds rule:

We used some of the research around Mechanical Turk to find best practices for limiting trolling (e.g. [1]). Our approach thus far has been the two-thirds rule: if two out of three people say the clip is good/bad, we trust that. Also note, that we have seen remarkable low trolling numbers, and that most of the invalid are pronunciation mistakes or saying the wrong word. 1.) https://groups.csail.mit.edu/sls/publications/2010/McGraw_LREC2010.pdf https://groups.csail.mit.edu/sls/publications/2010/McGraw_LR...