4 ms·
Ask HN: Does anyone use the Amazon Mechanical Turk API?
I'm working on a project that relies on a human workforce to perform some simple image identification tasks. AWS Turk seemed to fit the requirements pretty well, and I have the basics working with externally-hosted tasks.
Does anyone here actually use Turk, though? The forum seems nearly abandoned (http://developer.amazonwebservices.com/connect/forum.jspa?forumID=11), and no Amazon employees have responded to any questions. The API has not had any updates in years. Is the Mechanical Turk now a walking corpse?
Read on for a description of the problem I've run into. Maybe a fellow enterprising HN reader dealt with something similar and found a solution. :)
I need to enforce that the same worker does not look at closely-related images. I can enforce that in my app, and basically show a task that says "please return this task and try some of our other tasks in a couple of hours". This is annoying, and drives up the "returned tasks" statistic for the workers. I would very much like to make Amazon filter out tasks for users who have run out of valid work in my system.
At first, it seemed that Turk qualifications fit the bill. Unfortunately, the Turk system does not support "negative" qualification, e.g., it cannot only assign tasks to workers who do not have a particular qualification. Amazon employees have not addressed this question from a year ago: http://developer.amazonwebservices.com/connect/thread.jspa?threadID=32744&tstart=0
Then, I tried blocking workers once they run of tasks, and unblocking them once work shows up. This does not affect task filtering, and instead tells the user about a block. Additionally, blocking may or may not have really bad side effects for workers, another question Amazon employees have ignored: http://developer.amazonwebservices.com/connect/thread.jspa?threadID=48851&tstart=0
Advice and suggestions welcome.
- camtarn 16y agoCheck out http://www.turkernation.com/ http://www.turkernation.com/ - still seems active.
- pgbovine 16y agoCheck out this research project from MIT ... it aims to provide a more sane interface to MTurk: http://groups.csail.mit.edu/uid/turkit/ http://groups.csail.mit.edu/uid/turkit/
- pedoh 16y agoI'm not familiar with AMT, but you might check out http://crowdflower.com http://crowdflower.com and see if they might be a good fit for your needs.
- raffi 16y agoI use it for http://www.feedbackarmy.com http://www.feedbackarmy.com and other tasks. I recommend using an ExternalQuestion. This will let you host the worker form on your own site. I recommend using JavaScript and whatever you want server side to let the user know whether they can work on your stuff or not. Maybe set a cookie with an ID so you can know what kinds of images they've looked at. Doing something on your end is your best bet. I like MTurk but the API is not so great and support does not exist. Good luck.
- gcv 16y agoThank you! I am already using ExternalQuestion forms and generating the exact questions at view time. In your experience (or opinion), workers do not object if a task just says "sorry, please return this HIT and try again later"?
- raffi 16y agoI don't know the answer because I haven't been in your situation (yet). I'm thinking of offering support for worker demographics and the scheme I described is how I plan to implement it. a. Do I know about the worker? Yes - go to c. b. survey the worker with some demo questions and store them c. Does the worker's demographics match what the user wants? Yes - let them through d. Prompt worker--sorry, you're cute and all, but not what this user wants.
- maccman 16y agoWe using ExternalQuestions with the Ruby API lib for Socialmod. We're pretty impressed the quality of work that comes from MTurk - questions get answered under 5 mins. All the logic is our side, especially the banning/blocking logic, which gives us a lot of control. As to your problem, why not create a new hit-type for each group of similar images, and then restrict worker access based on that type? http://www.socialmod.com http://www.socialmod.com
- gcv 16y agoThanks, that promising and helpful. How do I restrict worker access by HIT type? I don't see anything in the documentation that lets me do this — negative qualifications would work, but the MTurk system does not support them. In your blocking logic, do you simply show a HIT that says "please return this HIT"? I already implemented this, but I'm worried that it is bad etiquette.
- jey 16y agoYou should have 1 task with many instances, and dynamically assign the specific task only after you have a specific user who has accepted it. This way you can use the user ID in your task-assigning logic.
- gcv 16y agoThat's exactly what I did. The question is, how do I prevent people who have no more valid tasks available from seeing or accepting HITs? I can lock them out in my own logic, but their Amazon records will show that they abandoned a task — and it wasn't actually their fault.
- lukejduncan 16y agoI know speakertext.com uses it, and for a cool product!