5 ms·

This article is an excellent summary of all the frustrations I've encountered using both oDesk and Mechanical Turk. I'll add one more issue with MTurk: binary

by tansey 15y ago

This article is an excellent summary of all the frustrations I've encountered using both oDesk and Mechanical Turk. I'll add one more issue with MTurk: binary decisions. Some of my tasks can't easily be broken down into accept/reject, or sometimes a worker gets something close but not all the way there. I want to be able to say "hey, you did pretty well, but please make these small changes and we'll accept it." But that's not possible right now. You get the HIT and it's accepted or rejected. The result is that we have to sometimes reject workers who are making an honest effort but didn't sufficiently complete the task or we have to accept sub-par answers so as to not have to deal with worker blowback.

Panos 15y ago
Let me suggest one solution for such cases: Use iterative tasks, in the spirit introduced by TurkIt. For example, you want to create a caption for an image. You let a user create a caption. Then you take this caption and give it to another user, asking the user to improve it. Take the two versions and ask other workers, "which of the two versions is better?". Iterate until no improvement is possible. Not a trivial setup, but gets around the binary accept/reject decisions problem and generates results of significantly superior quality.
- 83457 15y ago
  Good idea. I've also thought about this for data entry/analysis type tasks. It is fully expected that a worker on any level won't be perfect, especially on these types of services. An approach I've considered is to have have 3 workers do the same task then compare results and take the majority answer which should be close to if not 100% accurate. Overtime you could even rate workers for accuracy and recruit/pay more if possible.
  - Panos 15y ago
    On a very selfish note, take a look at these: http://qmturk.appspot.com/ http://qmturk.appspot.com/ http://code.google.com/p/get-another-label/ http://code.google.com/p/get-another-label/ You may find the code useful for what you are trying to do.
4clicknet 15y ago
In addition to the iterative approach, you can also offer a base rate and accept all the tasks, even if they're not quite perfect, then pay a bonus to the workers who really nailed it. That way you're not rejecting the workers who are making an honest effort, but the better performers are being paid accordingly.

by tansey 15y ago

Panos 15y ago

Let me suggest one solution for such cases: Use iterative tasks, in the spirit introduced by TurkIt. For example, you want to create a caption for an image. You let a user create a caption. Then you take this caption and give it to another user, asking the user to improve it. Take the two versions and ask other workers, "which of the two versions is better?". Iterate until no improvement is possible. Not a trivial setup, but gets around the binary accept/reject decisions problem and generates results of significantly superior quality.

83457 15y ago
Good idea. I've also thought about this for data entry/analysis type tasks. It is fully expected that a worker on any level won't be perfect, especially on these types of services. An approach I've considered is to have have 3 workers do the same task then compare results and take the majority answer which should be close to if not 100% accurate. Overtime you could even rate workers for accuracy and recruit/pay more if possible.
- Panos 15y ago
  On a very selfish note, take a look at these: http://qmturk.appspot.com/ http://qmturk.appspot.com/ http://code.google.com/p/get-another-label/ http://code.google.com/p/get-another-label/ You may find the code useful for what you are trying to do.

4clicknet 15y ago

In addition to the iterative approach, you can also offer a base rate and accept all the tasks, even if they're not quite perfect, then pay a bonus to the workers who really nailed it. That way you're not rejecting the workers who are making an honest effort, but the better performers are being paid accordingly.