9 ms·

Who Makes Below Minimum Wage in the Mechanical Turk Sweatshop?

80 points by narad 13y ago

incision 13y ago
I've been fascinated by the mturk concept since it launched. I sign in from time to time and complete tasks that look interesting. Unfortunately, the service is flooded with $0.02 spam tasks (Register an account and post ad-copy on a specific forum etc). The highest paying single tasks routinely involve translation/transcription of Arabic. The most I made from a single task was about $2.30 for 10 minutes, part of which was completion of a pre-screening task to demonstrate a certain level of English reading comprehension.
- _delirium 13y ago
  I used to do tasks now and then, but it feels like Amazon abandoned the platform. Early on there were more interesting tasks, engagement by the team, even improvements now and then. Now it's full of spam and scammers that Amazon doesn't seem to be doing anything about, and the platform doesn't seem to have had any improvements made since 2008 or so. To the extent it still works at all, it's because there's a subculture of "turkers" who use 3rd-party message boards and browser addons to navigate the mess.
- sliverstorm 13y ago
  The main thing that keeps me from engaging in mturk is finding tasks! I too like the concept, but I too do not speak Arabic and will not spam for pay. There were some transcriptionist tasks that looked interesting, but the pre-screening task was looking for professional transcriptionist level performance. Minimum qualifying WPM was like 70, and the acceptable error rate per minute was... zero? One? I'm not that good.
  - pc86 13y ago
    70 seems like a fairly reasonable rate to select for. If you've been touch typing for any measurable number of years I would think you'd at least be in the 70-80 range.
    - sliverstorm 13y ago
      I don't know about you, but while I can type at 70WPM for typing tests, I generally can't sustain it for long periods of time, and my error rate increases.
karolisd 13y ago
Like Airbnb takes advantage of a previously untapped supply of places to sleep, like Uber takes advantage of a previously untapped supply of driver's time, Mechanical Turk takes advantage of a previously untapped supply of people's minds. But is it as profitable? Why isn't there something that makes it super simple for me to take surveys on my phone while I wait somewhere? I'd rather answer some marketing questions for a quarter than play Dots as I wait to meet a friend.
- ctdonath 13y ago
  Looks like there's an opportunity to sell an app for this. A key for leveraging the untapped brainpower is presenting Mechanical Turk work in a way akin to many successful games: optimize the interface to a mobile device, make it "instant on" with pre-filtered options (no/minimal login & other logistics), just present the next task as something to do in a minute or so (or skip in a second). Sell that for a buck or $5, pushed as easy for a buyer to earn back fast. Crack for intellectuals?
  - FLUX-YOU 13y ago
    The desktop interface ought to be optimized as well. You could even go as far as bringing together multiple sites like M-turk to keep your filtered job queue full (standardized API, anyone?) It should also be possible to write something to check a bunch of bitcoin faucet sites and present tasks as they come available (though I suspect that's less than honest work, heh). Many of those have timers, which are easy to predict, and video watching can be put in the background so you can do things in parallel.
- incision 13y ago
  *>"Why isn't there something that makes it super simple for me to take surveys on my phone while I wait somewhere?" Judging by the going rates on mturk, you aren't going to make a quarter for a simple survey, more like a few cents. In my experience, a task that pays $0.25 likely to take 10+ minutes and require some actual effort. Not that these are bad things necessarily. I've seen some pattern-recognition type tasks on mturk in the $0.25 range that I'd call fun. Also, in the case of actual surveys I wonder how the value of a regular survey taker's responses would change over time? Would it go up by creating an ever more specific profile of yourself or go down as the collector already has data for your demographic? Are the surveys even necessary? Could someone simply create an open market for personal data? Just put some subset of your current snapshot up for bid and collect small passive income from accesses to it cutting out the middlemen who currently collect and market this data via their "free" web services. Fascinating possibilities, imo.
danielharan 13y ago
Exactly right: concerns about minimum wage are misplaced when the workers are foreigners. CloudFactory in Nepal pays workers about $1/hour (disclaimer: visited them, think they're cool, no other affiliation). Terrible for US workers, more than being a high school teacher in Nepal. What concerns me more is how all these systems are difficult to use and built for humans to complete tasks. The next revolution should be letting programmers use AIs.
- sliverstorm 13y ago
  The next revolution should be letting programmers use AIs. Because the only thing stopping them from using AI is that they aren't allowed to?
  - danielharan 13y ago
    I should have been more precise by adding in that marketplace. Last I checked, MTurk disallowed non-human workers, and that's holding back a lot of potential progress, IMNSHO.
  - Zircom 13y ago
    I think he's referring to how mturk and other things like it don't allow the use of AIs, it has to be an actual person.
ck2 13y ago
I don't think it's just companies online who use "independent contractors" to dodge minimum wage. I believe I've seen it firsthand at Walmart where certain staff are replaced by contracted workers. Next year the big scam is going to be everyone only works 29 hours. Because at the 30th hour the corporation has to contribute towards health care.
- acheron 13y ago
  Not sure how the 29-hour thing is a "scam". More like a "completely foreseeable consequence and understandable policy".
  - zzzeek 13y ago
    how do you explain the fact that this was not observed with Massachusetts' very similar health care system? http://www.urban.org/publications/412583.html http://www.urban.org/publications/412583.html
    - yummyfajitas 13y ago
      According to your document, the reason is that MA's employer penalty is about 7-10x smaller ($295) than the Obamacare employer penalty (either $2000 or $3000).
      - zzzeek 13y ago
        The document refers to this disparity but hypothesizes the opposite conclusion based on the fact that fewer firms would be impacted: > Although there are differences in the details, with penalties for firms that fail to comply higher under the ACA than in Massachusetts but fewer firms likely to be affected nationally, the broad similarities between the ACA and Massachusetts’ reform suggest that we can expect to see patterns in the response by employers under the ACA similar to those observed under health reform in Massachusetts.
    - ck2 13y ago
      That PDF shows nothing about full-time vs part-time downgrading. It only shows employment numbers were not reduced. In fact by reducing everyone to 29-hours, they could in theory claim to hire MORE people, while carefully saying nothing about them making a living wage or the state paying for their insurance vs the employer. That PDF is actually depressing when you look at the graphs, uninsurance was only reduced by a little over a third for all that change in policy.
jgh 13y ago
Last time I checked you could do pretty good with the audio transcription tasks if you have the knack for it (minimal stopping/replaying parts). The surveys usually aren't worth it but sometimes you'll get one that's only a couple of questions..
hooande 13y ago
Mechanical Turk and its ilk are very important. One of the biggest barriers to achievements in machine learning and optimization is a lack of processed and labelled data. A lot of statistical inference is about algorithms and formulae, but in most cases the key issue is having enough data. It doesn't matter how fancy your technology is if you don't have lots and lots of clearly labelled samples. Mechanical Turk (and crowdflower and a few other services) are important bridges for this. People can do important labeling and cleaning work for a cost that's affordable enough to be practical. Right now most of the work is translation, address finding or quasi spammy tasks. But the service itself has a lot of potential to do things that impact our lives in the future. Mechanical Turk workers can improve startups by helping with personalization and automation tasks, help with science and defense by labeling images and aid in all kinds of language processing and computer vision tasks that could make things cheaper and better for all of us. It's good to see human capital being utilized, even if it is ont he cheap. For what it's worth, amazon does recommend that people set the pricing on their mturk tasks so that they come out to something close to minimum wage. Though I'm not sure that things like this are what the minimum wage was designed for. Completing online labeling tasks is a far cry from working in an actual sweat shop. It would be nice if there was some sort of exception for technical piece work. Mechanical Turk isn't nearly popular enough right now, but in the future as machine learning becomes more common there will be a serious demand for these tasks. I hope we can find a way to let people do data preparation work while avoiding virtual dickensian conditions of a kind of data processing industrial revolution.
- flyinRyan 13y ago
  > For what it's worth, amazon does recommend that people set the pricing on their mturk tasks so that they come out to something close to minimum wage What is the incentive to do an online job and set your prices at the lowest possible legal limit? Surely nearly anything else you could do with a computer would be more valuable? I mean, if you get good enough playing computer games you can make more than minimum wage.
  - jiggy2011 13y ago
    It's the difference between a guaranteed payout right now vs a possible payout down the road. In order to get paid for online games you probably have to practice for quite a few hours for free before you might get good enough to be compensated.
deleted 13y ago
[deleted]
mkoble11 13y ago
While on the subject of mturk, you might find this useful: "Running Experiments Online Using Amazon Mechanical Turk" https://sites.google.com/a/cognitivescience.co/knowledgebase/resources/crowdsourcing-participants-and-work-using-amazon-mechanical-turk https://sites.google.com/a/cognitivescience.co/knowledgebase... Found it the other day via Steve Blank's amazing list of startup tools/resources: http://steveblank.com/tools-and-blogs-for-entrepreneurs/ http://steveblank.com/tools-and-blogs-for-entrepreneurs/
nawitus 13y ago
Has anyone made an AI that can complete some of the tasks?
- shurcooL 13y ago
  I wanted to, but they specifically forbid this. As the article mentions, the tasks are supposed to be those which cannot be easily automated and require the attention of a human.
  - nawitus 13y ago
    What's the reason to forbit automation? If they cannot be automated, then they won't be.
    - xlevus 13y ago
      Mturk can be used as a source for automation training. If you start writing bots to do the tasks, you're now the mostly-blind leading the blind.
      - dasmoth 13y ago
        That could be addressed by having an optional meatbags-only flag when creating tasks.
        jiggy2011 13y ago
        I guess it may also be for quality. For example it's not inconceivable that one could write an audio transcription bot that is quite good and may be good enough for a lot of tasks, but still not as good as a human and may fail quite badly at complex tasks. Once the market gets flooded with bots, the buyers will work out what's going on and start to lower their offers to cover the risk of getting a bot transcription. Then the real humans leave. And of course, if a task can be fully automated then amazon would prefer to integrate it into their cloud offering and take all of the profit themselves.
cm2012 13y ago
You can do minimum wage if you write proper english for some of the description stuff.
- nfailor 13y ago
  that was an underpaid job ten years ago when it was called technical writing. barely above minimum in mturk is a sad direction.
  - sliverstorm 13y ago
    The saddest part is the consequences. The quality of technical writing has fallen dramatically over the years.
jrokisky 13y ago
I did Mturk for a few months while traveling. Occasionally I was able to make ~7.00 an hour, but the average was closer to 2.50. The work is mind-numbing and underpaid.
- pc86 13y ago
  If it's mind-numbing, doesn't that by definition mean it's not worth paying very well?
  - e12e 13y ago
    Standing watch is mind numbing, but can be associated with risk - and isn't always paid terribly low.
leot 13y ago
Concerns are misplaced for many reasons. 1) no supervision 2) no dependency on any particular employer 3) ability to choose what to work on 4) ability to choose own hours 5) intrinsic trade-off between job compensation and the speed at which the job will be completed 6) no requirement that the worker be doing the task exclusively -- the could be watching TV, or chatting with a friend, or playing a video game 7) No possibility of employer coercion Here's another scenario: say you were paid to watch ads. Should that be subject to minimum wage laws? Well, clearly not: we're routinely paid $0 for the ads we're subjected to.
- e12e 13y ago
  I'm not sure it's accurate to say that we're not paid to watch ads -- we're not paid a salary -- but often we're given something else. Eg: substantially reduced prices on magazines, newspapers, tv etc (compared to if the sellers were to attempt to maintain a similar margin without ads). Personally I'm in favour of higher prices and less ads and product placement -- but to say that we're not paid to watch ads sounds a little disingenuous (while technically true).
n3rdy 13y ago
You may not make minimum wage recycling cans either, so lets shut down those bottle and can returns too!

80 points by narad 13y ago

I've been fascinated by the mturk concept since it launched. I sign in from time to time and complete tasks that look interesting. Unfortunately, the service is flooded with $0.02 spam tasks (Register an account and post ad-copy on a specific forum etc). The highest paying single tasks routinely involve translation/transcription of Arabic. The most I made from a single task was about $2.30 for 10 minutes, part of which was completion of a pre-screening task to demonstrate a certain level of English reading comprehension.

_delirium 13y ago
I used to do tasks now and then, but it feels like Amazon abandoned the platform. Early on there were more interesting tasks, engagement by the team, even improvements now and then. Now it's full of spam and scammers that Amazon doesn't seem to be doing anything about, and the platform doesn't seem to have had any improvements made since 2008 or so. To the extent it still works at all, it's because there's a subculture of "turkers" who use 3rd-party message boards and browser addons to navigate the mess.
sliverstorm 13y ago
The main thing that keeps me from engaging in mturk is finding tasks! I too like the concept, but I too do not speak Arabic and will not spam for pay. There were some transcriptionist tasks that looked interesting, but the pre-screening task was looking for professional transcriptionist level performance. Minimum qualifying WPM was like 70, and the acceptable error rate per minute was... zero? One? I'm not that good.
- pc86 13y ago
  70 seems like a fairly reasonable rate to select for. If you've been touch typing for any measurable number of years I would think you'd at least be in the 70-80 range.
  - sliverstorm 13y ago
    I don't know about you, but while I can type at 70WPM for typing tests, I generally can't sustain it for long periods of time, and my error rate increases.

karolisd 13y ago

Like Airbnb takes advantage of a previously untapped supply of places to sleep, like Uber takes advantage of a previously untapped supply of driver's time, Mechanical Turk takes advantage of a previously untapped supply of people's minds. But is it as profitable? Why isn't there something that makes it super simple for me to take surveys on my phone while I wait somewhere? I'd rather answer some marketing questions for a quarter than play Dots as I wait to meet a friend.

ctdonath 13y ago
Looks like there's an opportunity to sell an app for this. A key for leveraging the untapped brainpower is presenting Mechanical Turk work in a way akin to many successful games: optimize the interface to a mobile device, make it "instant on" with pre-filtered options (no/minimal login & other logistics), just present the next task as something to do in a minute or so (or skip in a second). Sell that for a buck or $5, pushed as easy for a buyer to earn back fast. Crack for intellectuals?
- FLUX-YOU 13y ago
  The desktop interface ought to be optimized as well. You could even go as far as bringing together multiple sites like M-turk to keep your filtered job queue full (standardized API, anyone?) It should also be possible to write something to check a bunch of bitcoin faucet sites and present tasks as they come available (though I suspect that's less than honest work, heh). Many of those have timers, which are easy to predict, and video watching can be put in the background so you can do things in parallel.
incision 13y ago
*>"Why isn't there something that makes it super simple for me to take surveys on my phone while I wait somewhere?" Judging by the going rates on mturk, you aren't going to make a quarter for a simple survey, more like a few cents. In my experience, a task that pays $0.25 likely to take 10+ minutes and require some actual effort. Not that these are bad things necessarily. I've seen some pattern-recognition type tasks on mturk in the $0.25 range that I'd call fun. Also, in the case of actual surveys I wonder how the value of a regular survey taker's responses would change over time? Would it go up by creating an ever more specific profile of yourself or go down as the collector already has data for your demographic? Are the surveys even necessary? Could someone simply create an open market for personal data? Just put some subset of your current snapshot up for bid and collect small passive income from accesses to it cutting out the middlemen who currently collect and market this data via their "free" web services. Fascinating possibilities, imo.

danielharan 13y ago

Exactly right: concerns about minimum wage are misplaced when the workers are foreigners. CloudFactory in Nepal pays workers about $1/hour (disclaimer: visited them, think they're cool, no other affiliation). Terrible for US workers, more than being a high school teacher in Nepal. What concerns me more is how all these systems are difficult to use and built for humans to complete tasks. The next revolution should be letting programmers use AIs.

sliverstorm 13y ago
The next revolution should be letting programmers use AIs. Because the only thing stopping them from using AI is that they aren't allowed to?
- danielharan 13y ago
  I should have been more precise by adding in that marketplace. Last I checked, MTurk disallowed non-human workers, and that's holding back a lot of potential progress, IMNSHO.
- Zircom 13y ago
  I think he's referring to how mturk and other things like it don't allow the use of AIs, it has to be an actual person.

ck2 13y ago

I don't think it's just companies online who use "independent contractors" to dodge minimum wage. I believe I've seen it firsthand at Walmart where certain staff are replaced by contracted workers. Next year the big scam is going to be everyone only works 29 hours. Because at the 30th hour the corporation has to contribute towards health care.

acheron 13y ago
Not sure how the 29-hour thing is a "scam". More like a "completely foreseeable consequence and understandable policy".
- zzzeek 13y ago
  how do you explain the fact that this was not observed with Massachusetts' very similar health care system? http://www.urban.org/publications/412583.html http://www.urban.org/publications/412583.html
  - yummyfajitas 13y ago
    According to your document, the reason is that MA's employer penalty is about 7-10x smaller ($295) than the Obamacare employer penalty (either $2000 or $3000).
    - zzzeek 13y ago
      The document refers to this disparity but hypothesizes the opposite conclusion based on the fact that fewer firms would be impacted: > Although there are differences in the details, with penalties for firms that fail to comply higher under the ACA than in Massachusetts but fewer firms likely to be affected nationally, the broad similarities between the ACA and Massachusetts’ reform suggest that we can expect to see patterns in the response by employers under the ACA similar to those observed under health reform in Massachusetts.
  - ck2 13y ago
    That PDF shows nothing about full-time vs part-time downgrading. It only shows employment numbers were not reduced. In fact by reducing everyone to 29-hours, they could in theory claim to hire MORE people, while carefully saying nothing about them making a living wage or the state paying for their insurance vs the employer. That PDF is actually depressing when you look at the graphs, uninsurance was only reduced by a little over a third for all that change in policy.

jgh 13y ago

Last time I checked you could do pretty good with the audio transcription tasks if you have the knack for it (minimal stopping/replaying parts). The surveys usually aren't worth it but sometimes you'll get one that's only a couple of questions..

hooande 13y ago

Mechanical Turk and its ilk are very important. One of the biggest barriers to achievements in machine learning and optimization is a lack of processed and labelled data. A lot of statistical inference is about algorithms and formulae, but in most cases the key issue is having enough data. It doesn't matter how fancy your technology is if you don't have lots and lots of clearly labelled samples. Mechanical Turk (and crowdflower and a few other services) are important bridges for this. People can do important labeling and cleaning work for a cost that's affordable enough to be practical. Right now most of the work is translation, address finding or quasi spammy tasks. But the service itself has a lot of potential to do things that impact our lives in the future. Mechanical Turk workers can improve startups by helping with personalization and automation tasks, help with science and defense by labeling images and aid in all kinds of language processing and computer vision tasks that could make things cheaper and better for all of us. It's good to see human capital being utilized, even if it is ont he cheap. For what it's worth, amazon does recommend that people set the pricing on their mturk tasks so that they come out to something close to minimum wage. Though I'm not sure that things like this are what the minimum wage was designed for. Completing online labeling tasks is a far cry from working in an actual sweat shop. It would be nice if there was some sort of exception for technical piece work. Mechanical Turk isn't nearly popular enough right now, but in the future as machine learning becomes more common there will be a serious demand for these tasks. I hope we can find a way to let people do data preparation work while avoiding virtual dickensian conditions of a kind of data processing industrial revolution.

flyinRyan 13y ago
> For what it's worth, amazon does recommend that people set the pricing on their mturk tasks so that they come out to something close to minimum wage What is the incentive to do an online job and set your prices at the lowest possible legal limit? Surely nearly anything else you could do with a computer would be more valuable? I mean, if you get good enough playing computer games you can make more than minimum wage.
- jiggy2011 13y ago
  It's the difference between a guaranteed payout right now vs a possible payout down the road. In order to get paid for online games you probably have to practice for quite a few hours for free before you might get good enough to be compensated.

deleted 13y ago

[deleted]

mkoble11 13y ago

While on the subject of mturk, you might find this useful: "Running Experiments Online Using Amazon Mechanical Turk" https://sites.google.com/a/cognitivescience.co/knowledgebase/resources/crowdsourcing-participants-and-work-using-amazon-mechanical-turk https://sites.google.com/a/cognitivescience.co/knowledgebase... Found it the other day via Steve Blank's amazing list of startup tools/resources: http://steveblank.com/tools-and-blogs-for-entrepreneurs/ http://steveblank.com/tools-and-blogs-for-entrepreneurs/

nawitus 13y ago

Has anyone made an AI that can complete some of the tasks?

shurcooL 13y ago
I wanted to, but they specifically forbid this. As the article mentions, the tasks are supposed to be those which cannot be easily automated and require the attention of a human.
- nawitus 13y ago
  What's the reason to forbit automation? If they cannot be automated, then they won't be.
  - xlevus 13y ago
    Mturk can be used as a source for automation training. If you start writing bots to do the tasks, you're now the mostly-blind leading the blind.
    - dasmoth 13y ago
      That could be addressed by having an optional meatbags-only flag when creating tasks.
      - jiggy2011 13y ago
        I guess it may also be for quality. For example it's not inconceivable that one could write an audio transcription bot that is quite good and may be good enough for a lot of tasks, but still not as good as a human and may fail quite badly at complex tasks. Once the market gets flooded with bots, the buyers will work out what's going on and start to lower their offers to cover the risk of getting a bot transcription. Then the real humans leave. And of course, if a task can be fully automated then amazon would prefer to integrate it into their cloud offering and take all of the profit themselves.

cm2012 13y ago

You can do minimum wage if you write proper english for some of the description stuff.

nfailor 13y ago
that was an underpaid job ten years ago when it was called technical writing. barely above minimum in mturk is a sad direction.
- sliverstorm 13y ago
  The saddest part is the consequences. The quality of technical writing has fallen dramatically over the years.

jrokisky 13y ago

I did Mturk for a few months while traveling. Occasionally I was able to make ~7.00 an hour, but the average was closer to 2.50. The work is mind-numbing and underpaid.

pc86 13y ago
If it's mind-numbing, doesn't that by definition mean it's not worth paying very well?
- e12e 13y ago
  Standing watch is mind numbing, but can be associated with risk - and isn't always paid terribly low.

leot 13y ago

Concerns are misplaced for many reasons. 1) no supervision 2) no dependency on any particular employer 3) ability to choose what to work on 4) ability to choose own hours 5) intrinsic trade-off between job compensation and the speed at which the job will be completed 6) no requirement that the worker be doing the task exclusively -- the could be watching TV, or chatting with a friend, or playing a video game 7) No possibility of employer coercion Here's another scenario: say you were paid to watch ads. Should that be subject to minimum wage laws? Well, clearly not: we're routinely paid $0 for the ads we're subjected to.

e12e 13y ago
I'm not sure it's accurate to say that we're not paid to watch ads -- we're not paid a salary -- but often we're given something else. Eg: substantially reduced prices on magazines, newspapers, tv etc (compared to if the sellers were to attempt to maintain a similar margin without ads). Personally I'm in favour of higher prices and less ads and product placement -- but to say that we're not paid to watch ads sounds a little disingenuous (while technically true).

n3rdy 13y ago

You may not make minimum wage recycling cans either, so lets shut down those bottle and can returns too!