2 ms·
I'm uncomfortable with research that merely states "we threw some Turkers at it". In my experience they are overall reliable, but there's no data on what qualif
by sehugg 11y ago
I'm uncomfortable with research that merely states "we threw some Turkers at it". In my experience they are overall reliable, but there's no data on what qualifications (if any) the jobs were assigned, how much was offered, how long was given to complete the test, etc. It could be that some of these humans were actually by bots themselves, if insufficient controls were put in place.
http://turkernation.com/showthread.php?21352-The-Myth-of-Low-Cost-High-Quality-on-Amazon-s-Mechanical-Turk http://turkernation.com/showthread.php?21352-The-Myth-of-Low...
- 51109 11y agoAn interesting paper at NIPS aims to combat the lower quality of Turkers with gamification. Instead of yes/no answers they offer a choice of: absolutely sure, yes, don't know, no, absolutely not. Monetary reward is doubled if they answer a question correctly with a high certainty, and it is zero'd when they are wrong. http://arxiv.org/abs/1408.1387 http://arxiv.org/abs/1408.1387 "Double or Nothing: Multiplicative Incentive Mechanisms for Crowdsourcing"