6 ms·

Amazon’s Mechanical Turk Has Reinvented Research

ehead 8y ago
It's an interesting question, whether or not the data is representative (or what demographic is the data representative of). Are there any studies done of the demographic distribution of Turkers?
- sideshowb 8y ago
  I'm told yes by my colleagues in political science. They seem to think highly of it. Don't have the citations myself sorry
- archgoon 8y ago
  Several of the referenced papers from the article seek to address this question: "Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk" https://www.jstor.org/stable/23260322?mag=amazons-mechanical-turk-has-reinvented-research&seq=1#page_scan_tab_contents https://www.jstor.org/stable/23260322?mag=amazons-mechanical... "Socially Mediated Internet Surveys: Recruiting Participants for Online Experiments" https://www.jstor.org/stable/43284764?mag=amazons-mechanical-turk-has-reinvented-research&seq=5#page_scan_tab_contents https://www.jstor.org/stable/43284764?mag=amazons-mechanical...
myro 8y ago
I've been turking once, made a $100 worth of Amazon credits and bought a first version of Kindle :) good old days
- dx034 8y ago
  How many hours? Was it 30-50 hours as some reports suggest or a decent hourly wage?
chx 8y ago
Data cleanup is also a great use. For example, we needed to parse an amount of address data, we just needed country, state (or equivalent), city, we gave out each address three times, whatever results were the same at least twice was accepted. We had over 92% where all three were the same, another 7% with 2-1 (required review), less than 1% needed either manual cleanup before re-Turking or just manually entering some of the more gnarly cases. We considered it a truly massive success, price efficient and absolutely unbelievable quick.
- sametmax 8y ago
  You outsourced personal data to mechanical turk ? Can I have the name of your company ?
  - jhall1468 8y ago
    Addresses are public data.
    - sametmax 8y ago
      It doesn't seem to be just addresses since they are extracted from something.
      - jamra 8y ago
        This comment makes no sense to me and it seems like you’re pushing this point with a lot of force. Of course there is context in how the addresses are used. But the reviewers may not have had access to the context. Since addresses are public data, the question remains.. what is the harm?
        Harvey-Specter 8y ago
        I've mentioned this in a few places but it seems to be ignored: The account that posts the addresses is assosciated with some company. If that company isn't just 'Generic Consulting Company' then there is some context derived from the account name. ie: This account is owned by 'XXX Toys', and they're asking me to clean up this address information. I guess someone at this address ordered some adult toys.
rococode 8y ago
MTurk is a godsend for ML research and is a huge game-changer. For every other project where the problem is "that sounds cool but we don't have enough labeled data" the answer nowadays is "just turk it". Sentiment labeling, qualitative comparison, error identification, and tons of other traditionally data-scarce tasks are made trivially easy (at the cost of some money) with MTurk, and it's pretty much a win-win for everyone involved too! Now the ethics as far as exploitation are definitely important, but I think the design of the site handles things quite well and makes everything fair for all parties. If you feel a task is underpaid, there are enough alternatives that you can just not do it. It's also true that there are many international turkers for whom $8/hour or less is still solid pay. Then there are also many third-party tools that allow turkers to see which HIT (task) requesters have good track records (low rejection ratio, good pay, etc.), and the site's own tools allow requesters to avoid turkers with bad track records. In my experience just browsing through tasks, heavily underpaid tasks don't tend to get done (for example, writing a 100-word summary for $0.50).
- krageon 8y ago
  You can't receive any actual money (only amazon money, which is much less useful) as a worker outside of the US as far as I know.
  - lozenge 8y ago
    You can in India as well which must be where most of the work is coming from.
- amelius 8y ago
  Except Amazon now also has access to your dataset.
  - brianwawok 8y ago
    If you are hosting your own data, Amazon knows the answer but not the question. So in say a photo classification.. they know "turk xxx said green" they don't know the photo. Seems like not too big of a deal, right? I can share with you all of my answer datasets, they are useless without the questions.
Radim 8y ago
Mechanical Turk is great for "open", public research. We used to use them a lot for machine learning tasks (data cleanup, model comparisons, label annotations), along with similar services like CrowdFlower / Figure Eight. We saw two primarily issues when applied to "non-open" (commercial) projects: - business-related data too sensitive to share with strangers (contractual obligations, too much risk) - some tasks required non-trivial subject matter expertise and context to annotate properly (quality control issues) For this reason, we gradually moved to an in-house team of long-term annotators. It's not much more expensive (moms on maternity leave, students…), but infinitely more flexible and safer for our purposes. YMMV.
- therealmarv 8y ago
  try PYBOSSA, open source crowdsourcing platform, you host it yourself, you are in control of your data.
  - k__ 8y ago
    I think the problem is more that you give the data to the turkers than to give it to Amazon.
- mrgordon 8y ago
  CrowdFlower / Figure Eight works with a lot of annotators who have signed NDAs and work in secure locations where they can't duplicate materials etc.
- mwexler 8y ago
  Seconded. Quality control is a nightmare with Turk, often requiring stimuli to be labeled multiple times and have a variety of judging approaches to "crown a winner". Companies like DefinedCrowd (https://www.definedcrowd.com/ https://www.definedcrowd.com/) have taken a quality-first approach which gives much better data, but of course, at a cost.
- ayw 8y ago
  Alex from Scale (www.scaleapi.com) here! We've taken an extremely quality-first approach and build out large workforces for datasets with high quality requirements and complexity. For example, we do a bunch of LIDAR / 3D labeling (https://www.scaleapi.com/sensor-fusion-annotation https://www.scaleapi.com/sensor-fusion-annotation) which is very complex and labor intensive, and provide extremely high quality that would not be possible otherwise.
Kyragem 8y ago
Maybe great for surveys but not a single HIT research project would be feasible if folks were required to pay minimum wage to their turkers.
jankotek 8y ago
Please fix misleading title, it should be: Reinvented _Survey_ Research. > thirty to forty minute survey ... paid $1.10 What sort of results do you expect from this "research"? Do you really expect people to read the questions and answer truthfully? You can prove pretty much anything, by reordering fields or manipulating the questions.
- glup 8y ago
  It can also be used for cognitive science research, for example to collect reading times, reaction times, vocabulary learning trajectories, etc. Published work suggests that it is provides data of similar quality to in-lab studies, once you do some due diligence (browser fingerprinting, etc.).
- mbym 8y ago
  Prolific (http://prolific.ac/ http://prolific.ac/) is designed specifically for science research, but I've used used it for two different use cases: 1. Product research, quickly testing appetite for an idea. 2. Micro tasks (a "survey" with at least one participant, where basically the only question was "have you done the task (on external page)?" The task in question was editing a public transcript, so confidentiality wasn't even a concern. Each time I was impressed with the quality of the responses. In the optional open-ended text fields for the research survey, in particular, I was impressed with how people took time to free-write their thoughts, rather than just rushing on to the next question / to complete the survey. Point is I do think you get what you pay for... From my experience, it seems Prolific's ethical approach (e.g., insisting on decent minimum payment rates) leads to overall higher quality of participants as well as responses (vs. race-to-the-bottom micro-task platforms like MTurk). [Disclaimer: I'm friends with the founders -- so I'm quite familiar with the amount of resources devoted to ensuring truthful/quality responses.]
ccnafr 8y ago
Do people who participate in MK get any money for their time?
- iagovar 8y ago
  Not outside the US, so I won't be so confident for non english data.
  - ccnafr 8y ago
    Why not? Are they afraid of users from low-income countries flooding the site?
    - mberger 8y ago
      Yes, and that you get a lot of tasks that you can't prosecute their origin. For example (off the top of my head) solving captchas, cracking passwords, doxing, hacking offers etc. I think there was also the spectre of organized crime enslaving people to do the tasks and taking the profits.
  - larrykwg 8y ago
    whaat non US workers don't get paid??
- deleted 8y ago
  [deleted]
ada1981 8y ago
My friend has used this extensively for all kinds of interesting things. I interviewed him about his work combining mturk & AI to help Trump / Clinton supporters better understand each other. https://huffpost.com/us/entry/us_581a4825e4b0f1c7d77c9555 https://huffpost.com/us/entry/us_581a4825e4b0f1c7d77c9555 With regard to pay, it seems reasonable to adopt a standard where the average per hour rate is disclosed in the research papers. This alone may provide social pressure for academics to adopt payment inline with local norms. Also this brings up some questions for me. Is it unethical for a researcher in a locale with low wages to post on mturk looking for work at comparable rates? Should posters be posting rates comparable to their own countries minimum wage laws? Is there another standard? Could requiring some researchers (from richer countries) to increase their rates result in resesrchers from other countries being priced out or having their research deprioritized by Turks?
knguyen0105 8y ago
Have anyone used Mechanical Turk for tasks that involve non-English languages like Chinese or Vietnamese?
- titanix2 8y ago
  I tried a batch for some data typing involving Chinese characters. Nobody did it. Maybe it wasn't paid enough, maybe there was no skilled people willing to do it. I'm also interested in other people experience involving to two languages.
  - jdietrich 8y ago
    That's not too surprising - most of Amazon's IP ranges are blocked by the Great Firewall.
jonbarker 8y ago
To paraphrase The Simpsons: "In the future, computer programs will be built by labeled data sets. And our job will be to build and maintain those labeled data sets."
Nasrudith 8y ago
I can't help but imagine unscrupulous bot programmers going through every possible survey and answering them quickly with garbage. Or if they monitor expected times doing parallel instances while waiting long enough to look like a human. High volume low value junk ruining things has a long history with the internet.
- benp84 8y ago
  Sadly there is disincentive to report bad workers on MTurk. You can block them through the MTurk system which rightfully puts their account at risk for termination. BUT Amazon sends them an email identifying you as the blocker, which causes them to go write terrible reviews of you on worker sites, which reduces the supply of workers for your HITs. So the ideal way to handle it is block them from your HIT server so at least they can't do YOUR HITs again and never report them.
8bitsrule 8y ago
'Mechanical Turk' is a sweatship run by a notorious skinflint.

It's an interesting question, whether or not the data is representative (or what demographic is the data representative of). Are there any studies done of the demographic distribution of Turkers?

sideshowb 8y ago
I'm told yes by my colleagues in political science. They seem to think highly of it. Don't have the citations myself sorry
archgoon 8y ago
Several of the referenced papers from the article seek to address this question: "Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk" https://www.jstor.org/stable/23260322?mag=amazons-mechanical-turk-has-reinvented-research&seq=1#page_scan_tab_contents https://www.jstor.org/stable/23260322?mag=amazons-mechanical... "Socially Mediated Internet Surveys: Recruiting Participants for Online Experiments" https://www.jstor.org/stable/43284764?mag=amazons-mechanical-turk-has-reinvented-research&seq=5#page_scan_tab_contents https://www.jstor.org/stable/43284764?mag=amazons-mechanical...

myro 8y ago

I've been turking once, made a $100 worth of Amazon credits and bought a first version of Kindle :) good old days

dx034 8y ago
How many hours? Was it 30-50 hours as some reports suggest or a decent hourly wage?

chx 8y ago

Data cleanup is also a great use. For example, we needed to parse an amount of address data, we just needed country, state (or equivalent), city, we gave out each address three times, whatever results were the same at least twice was accepted. We had over 92% where all three were the same, another 7% with 2-1 (required review), less than 1% needed either manual cleanup before re-Turking or just manually entering some of the more gnarly cases. We considered it a truly massive success, price efficient and absolutely unbelievable quick.

sametmax 8y ago
You outsourced personal data to mechanical turk ? Can I have the name of your company ?
- jhall1468 8y ago
  Addresses are public data.
  - sametmax 8y ago
    It doesn't seem to be just addresses since they are extracted from something.
    - jamra 8y ago
      This comment makes no sense to me and it seems like you’re pushing this point with a lot of force. Of course there is context in how the addresses are used. But the reviewers may not have had access to the context. Since addresses are public data, the question remains.. what is the harm?
      - Harvey-Specter 8y ago
        I've mentioned this in a few places but it seems to be ignored: The account that posts the addresses is assosciated with some company. If that company isn't just 'Generic Consulting Company' then there is some context derived from the account name. ie: This account is owned by 'XXX Toys', and they're asking me to clean up this address information. I guess someone at this address ordered some adult toys.

rococode 8y ago

MTurk is a godsend for ML research and is a huge game-changer. For every other project where the problem is "that sounds cool but we don't have enough labeled data" the answer nowadays is "just turk it". Sentiment labeling, qualitative comparison, error identification, and tons of other traditionally data-scarce tasks are made trivially easy (at the cost of some money) with MTurk, and it's pretty much a win-win for everyone involved too! Now the ethics as far as exploitation are definitely important, but I think the design of the site handles things quite well and makes everything fair for all parties. If you feel a task is underpaid, there are enough alternatives that you can just not do it. It's also true that there are many international turkers for whom $8/hour or less is still solid pay. Then there are also many third-party tools that allow turkers to see which HIT (task) requesters have good track records (low rejection ratio, good pay, etc.), and the site's own tools allow requesters to avoid turkers with bad track records. In my experience just browsing through tasks, heavily underpaid tasks don't tend to get done (for example, writing a 100-word summary for $0.50).

krageon 8y ago
You can't receive any actual money (only amazon money, which is much less useful) as a worker outside of the US as far as I know.
- lozenge 8y ago
  You can in India as well which must be where most of the work is coming from.
amelius 8y ago
Except Amazon now also has access to your dataset.
- brianwawok 8y ago
  If you are hosting your own data, Amazon knows the answer but not the question. So in say a photo classification.. they know "turk xxx said green" they don't know the photo. Seems like not too big of a deal, right? I can share with you all of my answer datasets, they are useless without the questions.

Radim 8y ago

Mechanical Turk is great for "open", public research. We used to use them a lot for machine learning tasks (data cleanup, model comparisons, label annotations), along with similar services like CrowdFlower / Figure Eight. We saw two primarily issues when applied to "non-open" (commercial) projects: - business-related data too sensitive to share with strangers (contractual obligations, too much risk) - some tasks required non-trivial subject matter expertise and context to annotate properly (quality control issues) For this reason, we gradually moved to an in-house team of long-term annotators. It's not much more expensive (moms on maternity leave, students…), but infinitely more flexible and safer for our purposes. YMMV.

therealmarv 8y ago
try PYBOSSA, open source crowdsourcing platform, you host it yourself, you are in control of your data.
- k__ 8y ago
  I think the problem is more that you give the data to the turkers than to give it to Amazon.
mrgordon 8y ago
CrowdFlower / Figure Eight works with a lot of annotators who have signed NDAs and work in secure locations where they can't duplicate materials etc.
mwexler 8y ago
Seconded. Quality control is a nightmare with Turk, often requiring stimuli to be labeled multiple times and have a variety of judging approaches to "crown a winner". Companies like DefinedCrowd (https://www.definedcrowd.com/ https://www.definedcrowd.com/) have taken a quality-first approach which gives much better data, but of course, at a cost.
ayw 8y ago
Alex from Scale (www.scaleapi.com) here! We've taken an extremely quality-first approach and build out large workforces for datasets with high quality requirements and complexity. For example, we do a bunch of LIDAR / 3D labeling (https://www.scaleapi.com/sensor-fusion-annotation https://www.scaleapi.com/sensor-fusion-annotation) which is very complex and labor intensive, and provide extremely high quality that would not be possible otherwise.

Kyragem 8y ago

Maybe great for surveys but not a single HIT research project would be feasible if folks were required to pay minimum wage to their turkers.

jankotek 8y ago

Please fix misleading title, it should be: Reinvented _Survey_ Research. > thirty to forty minute survey ... paid $1.10 What sort of results do you expect from this "research"? Do you really expect people to read the questions and answer truthfully? You can prove pretty much anything, by reordering fields or manipulating the questions.

glup 8y ago
It can also be used for cognitive science research, for example to collect reading times, reaction times, vocabulary learning trajectories, etc. Published work suggests that it is provides data of similar quality to in-lab studies, once you do some due diligence (browser fingerprinting, etc.).
mbym 8y ago
Prolific (http://prolific.ac/ http://prolific.ac/) is designed specifically for science research, but I've used used it for two different use cases: 1. Product research, quickly testing appetite for an idea. 2. Micro tasks (a "survey" with at least one participant, where basically the only question was "have you done the task (on external page)?" The task in question was editing a public transcript, so confidentiality wasn't even a concern. Each time I was impressed with the quality of the responses. In the optional open-ended text fields for the research survey, in particular, I was impressed with how people took time to free-write their thoughts, rather than just rushing on to the next question / to complete the survey. Point is I do think you get what you pay for... From my experience, it seems Prolific's ethical approach (e.g., insisting on decent minimum payment rates) leads to overall higher quality of participants as well as responses (vs. race-to-the-bottom micro-task platforms like MTurk). [Disclaimer: I'm friends with the founders -- so I'm quite familiar with the amount of resources devoted to ensuring truthful/quality responses.]

ccnafr 8y ago

Do people who participate in MK get any money for their time?

iagovar 8y ago
Not outside the US, so I won't be so confident for non english data.
- ccnafr 8y ago
  Why not? Are they afraid of users from low-income countries flooding the site?
  - mberger 8y ago
    Yes, and that you get a lot of tasks that you can't prosecute their origin. For example (off the top of my head) solving captchas, cracking passwords, doxing, hacking offers etc. I think there was also the spectre of organized crime enslaving people to do the tasks and taking the profits.
- larrykwg 8y ago
  whaat non US workers don't get paid??
deleted 8y ago
[deleted]

ada1981 8y ago

My friend has used this extensively for all kinds of interesting things. I interviewed him about his work combining mturk & AI to help Trump / Clinton supporters better understand each other. https://huffpost.com/us/entry/us_581a4825e4b0f1c7d77c9555 https://huffpost.com/us/entry/us_581a4825e4b0f1c7d77c9555 With regard to pay, it seems reasonable to adopt a standard where the average per hour rate is disclosed in the research papers. This alone may provide social pressure for academics to adopt payment inline with local norms. Also this brings up some questions for me. Is it unethical for a researcher in a locale with low wages to post on mturk looking for work at comparable rates? Should posters be posting rates comparable to their own countries minimum wage laws? Is there another standard? Could requiring some researchers (from richer countries) to increase their rates result in resesrchers from other countries being priced out or having their research deprioritized by Turks?

knguyen0105 8y ago

Have anyone used Mechanical Turk for tasks that involve non-English languages like Chinese or Vietnamese?

titanix2 8y ago
I tried a batch for some data typing involving Chinese characters. Nobody did it. Maybe it wasn't paid enough, maybe there was no skilled people willing to do it. I'm also interested in other people experience involving to two languages.
- jdietrich 8y ago
  That's not too surprising - most of Amazon's IP ranges are blocked by the Great Firewall.

jonbarker 8y ago

To paraphrase The Simpsons: "In the future, computer programs will be built by labeled data sets. And our job will be to build and maintain those labeled data sets."

Nasrudith 8y ago

I can't help but imagine unscrupulous bot programmers going through every possible survey and answering them quickly with garbage. Or if they monitor expected times doing parallel instances while waiting long enough to look like a human. High volume low value junk ruining things has a long history with the internet.

benp84 8y ago
Sadly there is disincentive to report bad workers on MTurk. You can block them through the MTurk system which rightfully puts their account at risk for termination. BUT Amazon sends them an email identifying you as the blocker, which causes them to go write terrible reviews of you on worker sites, which reduces the supply of workers for your HITs. So the ideal way to handle it is block them from your HIT server so at least they can't do YOUR HITs again and never report them.

8bitsrule 8y ago

'Mechanical Turk' is a sweatship run by a notorious skinflint.