3 ms·
Most of it boils down to trust. The main problems of these systems are that if you can get $3/hr of economic value from a single turk, it's hard to trust that
by ukd1 14y ago
Most of it boils down to trust.
The main problems of these systems are that if you can get $3/hr of economic value from a single turk, it's hard to trust that you are getting accurate results without checking by repetition. This means that you could end up repeating the task 2, 3 times or more to get accuracy.
Scammers and bad workers are a big issue on any platform like this. I'm not sure how easily this can be solved - the workers only incentive to not cheat the system is the possibility of the requester rejecting their work and them not getting paid as well damaging their rating.
Whilst this seems like a good-enough way, there are enough lazy / bad requesters who don't bother rejecting work which skews ratings of workers. Also, ratings on mturk are lifetime ratings, rather than last n days, which isn't that indicative of recent worker quality.
- anandkulkarni 14y agoI'd argue the solution to trust is appropriately matching work to reliable workers for whom the work is fairly paid, not simply raw repetition. At $3 an hour, someone in (say) Lithuania would be highly motivated to do the work right, even if someone in the US wouldn't particularly care. This is the approach we took at MobileWorks, and it works fairly well.
- sage_joch 14y agoA possible solution could be for users to have an accuracy rating (perhaps short-term and long-term), and for the Turk API to take an optional minimum accuracy rating for tasks to be visible to a given user. (For all I know this is already in place; I'm not a Turk user.)