3 ms·
I have used Mechanical Turk quite extensively the last few months. But I have always used it via Crowdflower. They offer a more sensible API and interface. And
by PanMan 16y ago
I have used Mechanical Turk quite extensively the last few months. But I have always used it via Crowdflower. They offer a more sensible API and interface. And they keep scores on all workers, and calculate how well they perform. This influcences the weight of their job submissions. It takes a bit getting used to their terms, but I'm quite happy with the end results.
- bravura 16y agoCrowdFlower has some annoying particularities which are undocumented. In particular: The CSV parser doesn't work / is too strict / something. The only way to upload complicated data files in as "JSON". The only problem is they don't accept standard JSON. You cannot have a comma separating the rows. (!?) I have no idea how much time I spent trying to figure this out. Additionally, if your task is too hard, and more than 50% of judgments are rejected, they auto-pause your task. When you resume it, it will simply pause a few minutes later. So, for certain tasks, you simply cannot finish the job. Their rationale is that CrowdFlower pays for the judgments that are discarded, and you pay for only the judgments they keep, so if they discard too many judgments they lose money. I would rather just pay extra money for tasks that are very hard.
- nmcfarl 16y agoCrowdflower's API is excellent in comparison. And the UI is much better to boot. We find it restricts the kinds of work we like to do too much - but it also restricts it to the kinds of work that Crowdflower can reasonably return good results on. So that seems like a good trade off. I recommend Crowdflower as a starting place to anyone thinking of using MTurk.