4 ms·

You set up the HIT the wrong way. The correct way to do this is to assign the same URL to multiple Turkers and look for cases where people disagree in order to

You set up the HIT the wrong way. The correct way to do this is to assign the same URL to multiple Turkers and look for cases where people disagree in order to verify the response quality. Granted it's still possible that both Turkers assigned to the HIT could commit fraud in the same way, but surely just as a matter of chance this bounds your error rate pretty substantially.

ceejayoz 8y ago
You can also use qualification tests (https://docs.aws.amazon.com/AWSMechTurk/latest/AWSMechanicalTurkRequester/Concepts_QualificationsArticle.html#qualification-tests https://docs.aws.amazon.com/AWSMechTurk/latest/AWSMechanical...) to require Turkers to demonstrate competence before doing tasks. I'd imagine you could inject a certain percentage of known-answer tasks into the queue as well, as ongoing quality control.
turtlegrids 8y ago
> Granted it's still possible that both Turkers assigned to the HIT could commit fraud in the same way, but surely just as a matter of chance this bounds your error rate pretty substantially. I wonder if there's been research into the effort in time or persons to perpetuate this fraud. Knowing Amazon, I'm certain they've analyzed this internally if not constantly at least a few times over the years. I wonder if there's a public attempt at the same analysis.