3 ms·

Unlike turkers, grad students produce much cleaner data and also see the data first hand. Which is important. I consider mturk/offshore as a necessary evil, rat

Unlike turkers, grad students produce much cleaner data and also see the data first hand. Which is important. I consider mturk/offshore as a necessary evil, rather than a panacea that will "make research easier and speed it up". If data quality matters and you can avoid mturk - avoid it.

prayag 14y ago
>Unlike turkers, grad students produce much cleaner data and also see the data first hand. Which is important. I consider mturk/offshore as a necessary evil, rather than a panacea that will "make research easier and speed it up". This is what we are trying to solve. You can get good clean data from crowdsourcing too if you give it out to the most suitable workers and have them work collaboratively.
anandkulkarni 14y ago
>If data quality matters and you can avoid mturk - avoid it. You've identified a great reason to use MobileWorks instead. Crowd work doesn't have to be low-quality; Turk just hasn't done a terrific job of it. We've achieved substantially higher quality than MTurk by treating workers better, paying fairer wages, and giving workers tools to collaborate.
huahaiy 14y ago
You get what you pay for. If you pay them pennies, you get garbage, but if you pay them equitable wages, you get real work done. It's that simple.
aantix 14y ago
Seems like you're painting with a broad brush. Design of HITs on Mechanical Turk is crucial. Implement gold standard questions per assigment to ensure that you're not receiving spam from a bot.
- anandkulkarni 14y ago
  It turns out this is one of the core problems with Mechanical Turk that drives folks to MobileWorks instead. In Turk, users spend so much time implementing protection against malicious workers and bots that they need to develop real expertise in crowdsourcing in order to be able to outsource their work effectively. We've filtered workers more effectively up front and reduced the incentive to cheat substantially, so things largely just work.

prayag 14y ago

>Unlike turkers, grad students produce much cleaner data and also see the data first hand. Which is important. I consider mturk/offshore as a necessary evil, rather than a panacea that will "make research easier and speed it up". This is what we are trying to solve. You can get good clean data from crowdsourcing too if you give it out to the most suitable workers and have them work collaboratively.

anandkulkarni 14y ago

>If data quality matters and you can avoid mturk - avoid it. You've identified a great reason to use MobileWorks instead. Crowd work doesn't have to be low-quality; Turk just hasn't done a terrific job of it. We've achieved substantially higher quality than MTurk by treating workers better, paying fairer wages, and giving workers tools to collaborate.

huahaiy 14y ago

You get what you pay for. If you pay them pennies, you get garbage, but if you pay them equitable wages, you get real work done. It's that simple.

aantix 14y ago

Seems like you're painting with a broad brush. Design of HITs on Mechanical Turk is crucial. Implement gold standard questions per assigment to ensure that you're not receiving spam from a bot.

anandkulkarni 14y ago
It turns out this is one of the core problems with Mechanical Turk that drives folks to MobileWorks instead. In Turk, users spend so much time implementing protection against malicious workers and bots that they need to develop real expertise in crowdsourcing in order to be able to outsource their work effectively. We've filtered workers more effectively up front and reduced the incentive to cheat substantially, so things largely just work.