3 ms·
MTurk is a godsend for ML research and is a huge game-changer. For every other project where the problem is "that sounds cool but we don't have enough labeled d
by rococode 8y ago
MTurk is a godsend for ML research and is a huge game-changer. For every other project where the problem is "that sounds cool but we don't have enough labeled data" the answer nowadays is "just turk it". Sentiment labeling, qualitative comparison, error identification, and tons of other traditionally data-scarce tasks are made trivially easy (at the cost of some money) with MTurk, and it's pretty much a win-win for everyone involved too!
Now the ethics as far as exploitation are definitely important, but I think the design of the site handles things quite well and makes everything fair for all parties. If you feel a task is underpaid, there are enough alternatives that you can just not do it. It's also true that there are many international turkers for whom $8/hour or less is still solid pay. Then there are also many third-party tools that allow turkers to see which HIT (task) requesters have good track records (low rejection ratio, good pay, etc.), and the site's own tools allow requesters to avoid turkers with bad track records. In my experience just browsing through tasks, heavily underpaid tasks don't tend to get done (for example, writing a 100-word summary for $0.50).
- krageon 8y agoYou can't receive any actual money (only amazon money, which is much less useful) as a worker outside of the US as far as I know.
- lozenge 8y agoYou can in India as well which must be where most of the work is coming from.
- amelius 8y agoExcept Amazon now also has access to your dataset.
- brianwawok 8y agoIf you are hosting your own data, Amazon knows the answer but not the question. So in say a photo classification.. they know "turk xxx said green" they don't know the photo. Seems like not too big of a deal, right? I can share with you all of my answer datasets, they are useless without the questions.
- amelius 8y agoThe turks access your data (the questions) through Amazon, and I don't see how this will happen without Amazon having access to that data.
- Harvey-Specter 8y agoThis doesn't have to be the case. For example, if you're paying mturk workers to fill out a survey you can give them a link to some third party (ie: Qualtrics) which will generate a code upon survey completion which the mturker enters as the answer to the only question on Amazon. If you're really concerned about a third party having access to your data, you could just host the survey yourself.
- amelius 8y agoOk, but Amazon also checks for turk performance, e.g. to keep out bots, and people who just submit wrong answers to make a quick buck. It doesn't seem trivial to build upon that by returning a code. (A related question: would it be possible to use Amazon Turk to train an algorithm that answers Amazon Turk questions?)
- Harvey-Specter 8y agoYou would still use mTurk to recruit participants, so you still benefit from their filters for acceptance rate etc. You would just implement the survey (probably with some attention checks to weed out people who just randomly answer questions), and at survey completion you give them a code to submit to complete the mTurk hit. Then you would only accept the submissions that have passed the attention checks, and pay those mTurkers.
- brianwawok 8y agoExternal Questions https://docs.aws.amazon.com/AWSMechTurk/latest/AWSMturkAPI/ApiReference_ExternalQuestionArticle.html https://docs.aws.amazon.com/AWSMechTurk/latest/AWSMturkAPI/A... This seems one of the more popular ways to do turks. You send people off to your own site, to show them stuff. All you really provide is a url, like www.mysite.com/questionId=123456... and render whatever you want when that appears
- Fnoord 8y ago> It's also true that there are many international turkers for whom $8/hour or less is still solid pay > and it's pretty much a win-win for everyone involved too! It is not; it is a way to pay below minimum wage, and to have simple labor paid to the lowest common denominator wage in the world. Hence people from countries such as Germany, The Netherlands, United Kingdom would be underpaid with $8 hour. The problem is that these people also have higher expenses than poor countries. My rent alone is 600+ EUR/month, and that's cheap (for NL).