7 ms·

Shortcomings of Amazon Mechanical Turk May Threaten Natural Language Generation

88 points by Hard_Space 5y ago

arglebarglegar 5y ago
this service always made me feel uncomfortable, even the origin of the name “mechanical turk”… an illusion of automation meant to fool others, but really just the hidden labor of a foreigner
- 01acheru 5y ago
  The name “mechanical turk” comes from a fake chess-playing machine that hid a person inside of it, whom actually operated said machine. So I think the name is quite appropriate for this service. Anyway, I don’t like that service either. https://en.m.wikipedia.org/wiki/Mechanical_Turk https://en.m.wikipedia.org/wiki/Mechanical_Turk
  - klyrs 5y ago
    I think GP knows that. The name is uncomfortable and it's honest. Which... kinda says something about the people using it for scientific research.
    - asciident 5y ago
      What does it say about people using it for research?
    - 01acheru 5y ago
      Why would it be uncomfortable? Those hidden inside the (original) machine were not exploited by the machine owner edit: I see that wearywanderer explained what I mean as a sibling, that’s why I think GP didn’t get precisely what the MT was and how it was operated
      - webmaven 5y ago
        > Why would it be uncomfortable? Those hidden inside the (original) machine were not exploited by the machine owner Do you have a source for that assertion? As far as I can tell, several of the chess players known to have operated the Turk had substance abuse (chiefly alcoholism), health, and money problems (the words 'debt', 'penniless', and 'destitute' come up a few times). While not proof of abuse, it does suggest a strategy of recruiting the vulnerable.
        01acheru 5y ago
        In late 1700 and early 1800 Europe being a heavy drinker, having some kind of health issue and/or being penniless were the norm for most of the population. If performing a task because of those issues means being exploited then we must assume that almost everyone in Europe at those times were exploited in one way or another. I understand what you mean if we judge it from modern western standards, but I don’t agree if we judge it by the standards of that era. It’s all a matter of context IMHO.
        deleted 5y ago
        [deleted]
- kongin 5y ago
  When the Mechanical Turk was crated in Austria the country had been nearly conquered by the Ottomans less than 80 years previously: https://en.wikipedia.org/wiki/Battle_of_Vienna https://en.wikipedia.org/wiki/Battle_of_Vienna The name was meant to inspire dread in those playing against it, much like how you know shit has gotten real when you have to fight robot Hitler in an American game.
- legrande 5y ago
  > an illusion of automation meant to fool others They even say it on the site: 'Artificial Artificial Intelligence'
- wearywanderer 5y ago
  > the hidden labor of a foreigner You're technically not wrong, but from your tone and wording I think you have a misconception about the machine, or are trying to induce such a misconception in others, to inspire others to take offense at the premise. There was never a Turkish person hidden inside the Mechanical Turk. The "foreigners" inside were a variety of chess masters from Germany, Austrian, French and the UK. Not the oppressed immigrants a modern reader might imagine when speaking of Turks and unspecified foreigners. And I suspect the chessmaster operators of the original Mechanical Turk were not remotely representative of the demographics of modern Amazon MT users. A Frenchman participating in a scheme to bamboozle some Austrian princes is not exactly something worth getting bent out of shape over. (Furthermore I think in at least some of the cases, the chessmasters were operating the machine in their home country and weren't foreigners at all.)
  - arglebarglegar 5y ago
    the turk operator got zero credit for beating heaps of famous people while inside the mecahnical turk, it’s hidden foreigner labor either way… but amazon reversed the exploitation aspect this is true
ulucs 5y ago
Why would you be surprised that you get monkeys when you're paying peanuts?
bravura 5y ago
Ever since CrowdFlower (now Figure8) decided to terminate its SaaS-offering and focus only on enterprise, Amazon Mechanical Turk has been impossible to use. CrowdFlower maintained statistics about the accuracy of individual workers. Additionally, they made it easy to include gold-standard "test" questions, which weeds out workers who are not doing the specific task correctly. Without this sort of quality-control platform, mechanical turk is just unreliable crowdsourced-labor infrastructure. I understand that AWS doesn't want to build too much on the services they provide but, really, mechanical turk sucks. I'm working with a researcher who is conducting experiments using turk, and he has a half-written shoddy version of what crowdflower offered. It seems that everyone using turk has to reinvent the wheel. Why can't someone provide decent quality control over turk results?
- imglorp 5y ago
  Sounds like a business opportunity? Quality/metrics layer as a Saas with MT on backend, add a few percent to the price per work unit?
  - kuzee 5y ago
    Sounds sweet. I bet a lot of companies relying on mturk built this for themselves and then sell a higher value service with better margins. You could build something right in the middle. I know Stanford's research teams all use a common interface to mturk that keeps profiles of turkers on their side so they know who to solicit for upcoming surveys, conduct longitudinal studies, etc. I've always wondered why more universities didn't follow suit. I built a side project called cogmint based on the insight that simple scoring and ranking of workers was valuable. I ended up building my own worker interface instead of using mturk because it wasn't much additional effort on top of the scoring logic I was building anyway. Perhaps other serious companies came to the same conclusion I did with my hobby project.
  - catillac 5y ago
    Sounds a little meta! Managed managed labeling workflow. I do wish Amazon would do a better job with this, and I also have no doubt if a company like this sprang into existence and was successful, Amazon would clamp down on it in some way — whether by changing terms to disallow such a business, cutting off MT access to the business, or just copying it.
curiousgal 5y ago
I've been saying this for years as a critique to psychology studies. Do you really expect desperate people getting paid peanuts to yield quality results?
- kongin 5y ago
  I'm sorry, but are we talking about the post docs ~running the experiment~ managing the routine tasks beneath the dignity of the department/lab head or the people participating in the experiment?
  - hervature 5y ago
    I am not privy to the state of psychology research, but I'm pretty sure the OP is referring to the reliance of Mechanical Turk in the field of psychology [1]. [1] - https://www.newscientist.com/article/2176436-bots-on-amazons-mechanical-turk-are-ruining-psychology-studies/ https://www.newscientist.com/article/2176436-bots-on-amazons...
- catillac 5y ago
  I mentioned this above, but at least in the short term my experience has been that paying more doesn’t yield better results on most tasks, which I naively attribute to those skills not being available at any price from the current population of MTers. If higher prices were paid across the board, doubtlessly more workers with the sought after skills would sign up though. So maybe that was the thrust of your comment.
  - joe_the_user 5y ago
    I mentioned this above, but at least in the short term my experience has been that paying more doesn’t yield better results on most tasks, which I naively attribute to those skills not being available at any price from the current population of MTers. This is plausible but doesn't change the basic problem. Of course, workers on AMT have years of experience trying to game it. Just more pay isn't going to create a sudden sense of solidarity with the employer. I would guess that quality results at scale would require a system built from the ground-up to cultivate some degree of loyalty or something. It's kind of a dilema of capitalism itself. At scale, you can pay people to accomplish a task for peanuts but they lack loyalty and will cut-corners whenever it's possible and convenient. And that still gets a whole lot of things done for cheap in this world. But if you want to get loyalty, people you don't have to watch, etc., you have to organize an entire enterprise for that, maybe show loyalty yourself (Turkers are inherently throw-away, why should they be loyal for just today's wages). And the cost of that arrangement tends to multiply.
geraldwhen 5y ago
The pay is so low that if you’re not an idiot, you’ll automatically fill out the form randomly. Any data from mturk cannot be trusted.
- georgeutsin 5y ago
  Even with more pay, there aren’t any real repercussions for doing bad work. Sure there are different “tiers” of turkers, but realistically anyone could recreate their account once their rating gets low enough.
gilrain 5y ago
When you aren’t willing to pay for quality labor, I wonder what you get?
legrande 5y ago
> AMT workers use VPNs to work around geographical restrictions Interesting. When I go to login to Amazon with a VPN it puts me in a CAPTCHA loop and never lets me access my account. I have to login with a residential IP address to access my account.
kderbyma 5y ago
The whole company was shady from the get go...it was effectively slave labour....it's Amazon's bread and butter of just disregarding the quality in favour of massive availability

88 points by Hard_Space 5y ago

this service always made me feel uncomfortable, even the origin of the name “mechanical turk”… an illusion of automation meant to fool others, but really just the hidden labor of a foreigner

01acheru 5y ago
The name “mechanical turk” comes from a fake chess-playing machine that hid a person inside of it, whom actually operated said machine. So I think the name is quite appropriate for this service. Anyway, I don’t like that service either. https://en.m.wikipedia.org/wiki/Mechanical_Turk https://en.m.wikipedia.org/wiki/Mechanical_Turk
- klyrs 5y ago
  I think GP knows that. The name is uncomfortable and it's honest. Which... kinda says something about the people using it for scientific research.
  - asciident 5y ago
    What does it say about people using it for research?
  - 01acheru 5y ago
    Why would it be uncomfortable? Those hidden inside the (original) machine were not exploited by the machine owner edit: I see that wearywanderer explained what I mean as a sibling, that’s why I think GP didn’t get precisely what the MT was and how it was operated
    - webmaven 5y ago
      > Why would it be uncomfortable? Those hidden inside the (original) machine were not exploited by the machine owner Do you have a source for that assertion? As far as I can tell, several of the chess players known to have operated the Turk had substance abuse (chiefly alcoholism), health, and money problems (the words 'debt', 'penniless', and 'destitute' come up a few times). While not proof of abuse, it does suggest a strategy of recruiting the vulnerable.
      - 01acheru 5y ago
        In late 1700 and early 1800 Europe being a heavy drinker, having some kind of health issue and/or being penniless were the norm for most of the population. If performing a task because of those issues means being exploited then we must assume that almost everyone in Europe at those times were exploited in one way or another. I understand what you mean if we judge it from modern western standards, but I don’t agree if we judge it by the standards of that era. It’s all a matter of context IMHO.
        deleted 5y ago
        [deleted]
kongin 5y ago
When the Mechanical Turk was crated in Austria the country had been nearly conquered by the Ottomans less than 80 years previously: https://en.wikipedia.org/wiki/Battle_of_Vienna https://en.wikipedia.org/wiki/Battle_of_Vienna The name was meant to inspire dread in those playing against it, much like how you know shit has gotten real when you have to fight robot Hitler in an American game.
legrande 5y ago
> an illusion of automation meant to fool others They even say it on the site: 'Artificial Artificial Intelligence'
wearywanderer 5y ago
> the hidden labor of a foreigner You're technically not wrong, but from your tone and wording I think you have a misconception about the machine, or are trying to induce such a misconception in others, to inspire others to take offense at the premise. There was never a Turkish person hidden inside the Mechanical Turk. The "foreigners" inside were a variety of chess masters from Germany, Austrian, French and the UK. Not the oppressed immigrants a modern reader might imagine when speaking of Turks and unspecified foreigners. And I suspect the chessmaster operators of the original Mechanical Turk were not remotely representative of the demographics of modern Amazon MT users. A Frenchman participating in a scheme to bamboozle some Austrian princes is not exactly something worth getting bent out of shape over. (Furthermore I think in at least some of the cases, the chessmasters were operating the machine in their home country and weren't foreigners at all.)
- arglebarglegar 5y ago
  the turk operator got zero credit for beating heaps of famous people while inside the mecahnical turk, it’s hidden foreigner labor either way… but amazon reversed the exploitation aspect this is true

ulucs 5y ago

Why would you be surprised that you get monkeys when you're paying peanuts?

bravura 5y ago

Ever since CrowdFlower (now Figure8) decided to terminate its SaaS-offering and focus only on enterprise, Amazon Mechanical Turk has been impossible to use. CrowdFlower maintained statistics about the accuracy of individual workers. Additionally, they made it easy to include gold-standard "test" questions, which weeds out workers who are not doing the specific task correctly. Without this sort of quality-control platform, mechanical turk is just unreliable crowdsourced-labor infrastructure. I understand that AWS doesn't want to build too much on the services they provide but, really, mechanical turk sucks. I'm working with a researcher who is conducting experiments using turk, and he has a half-written shoddy version of what crowdflower offered. It seems that everyone using turk has to reinvent the wheel. Why can't someone provide decent quality control over turk results?

imglorp 5y ago
Sounds like a business opportunity? Quality/metrics layer as a Saas with MT on backend, add a few percent to the price per work unit?
- kuzee 5y ago
  Sounds sweet. I bet a lot of companies relying on mturk built this for themselves and then sell a higher value service with better margins. You could build something right in the middle. I know Stanford's research teams all use a common interface to mturk that keeps profiles of turkers on their side so they know who to solicit for upcoming surveys, conduct longitudinal studies, etc. I've always wondered why more universities didn't follow suit. I built a side project called cogmint based on the insight that simple scoring and ranking of workers was valuable. I ended up building my own worker interface instead of using mturk because it wasn't much additional effort on top of the scoring logic I was building anyway. Perhaps other serious companies came to the same conclusion I did with my hobby project.
- catillac 5y ago
  Sounds a little meta! Managed managed labeling workflow. I do wish Amazon would do a better job with this, and I also have no doubt if a company like this sprang into existence and was successful, Amazon would clamp down on it in some way — whether by changing terms to disallow such a business, cutting off MT access to the business, or just copying it.

curiousgal 5y ago

I've been saying this for years as a critique to psychology studies. Do you really expect desperate people getting paid peanuts to yield quality results?

kongin 5y ago
I'm sorry, but are we talking about the post docs ~running the experiment~ managing the routine tasks beneath the dignity of the department/lab head or the people participating in the experiment?
- hervature 5y ago
  I am not privy to the state of psychology research, but I'm pretty sure the OP is referring to the reliance of Mechanical Turk in the field of psychology [1]. [1] - https://www.newscientist.com/article/2176436-bots-on-amazons-mechanical-turk-are-ruining-psychology-studies/ https://www.newscientist.com/article/2176436-bots-on-amazons...
catillac 5y ago
I mentioned this above, but at least in the short term my experience has been that paying more doesn’t yield better results on most tasks, which I naively attribute to those skills not being available at any price from the current population of MTers. If higher prices were paid across the board, doubtlessly more workers with the sought after skills would sign up though. So maybe that was the thrust of your comment.
- joe_the_user 5y ago
  I mentioned this above, but at least in the short term my experience has been that paying more doesn’t yield better results on most tasks, which I naively attribute to those skills not being available at any price from the current population of MTers. This is plausible but doesn't change the basic problem. Of course, workers on AMT have years of experience trying to game it. Just more pay isn't going to create a sudden sense of solidarity with the employer. I would guess that quality results at scale would require a system built from the ground-up to cultivate some degree of loyalty or something. It's kind of a dilema of capitalism itself. At scale, you can pay people to accomplish a task for peanuts but they lack loyalty and will cut-corners whenever it's possible and convenient. And that still gets a whole lot of things done for cheap in this world. But if you want to get loyalty, people you don't have to watch, etc., you have to organize an entire enterprise for that, maybe show loyalty yourself (Turkers are inherently throw-away, why should they be loyal for just today's wages). And the cost of that arrangement tends to multiply.

geraldwhen 5y ago

The pay is so low that if you’re not an idiot, you’ll automatically fill out the form randomly. Any data from mturk cannot be trusted.

georgeutsin 5y ago
Even with more pay, there aren’t any real repercussions for doing bad work. Sure there are different “tiers” of turkers, but realistically anyone could recreate their account once their rating gets low enough.

gilrain 5y ago

When you aren’t willing to pay for quality labor, I wonder what you get?

legrande 5y ago

> AMT workers use VPNs to work around geographical restrictions Interesting. When I go to login to Amazon with a VPN it puts me in a CAPTCHA loop and never lets me access my account. I have to login with a residential IP address to access my account.

kderbyma 5y ago

The whole company was shady from the get go...it was effectively slave labour....it's Amazon's bread and butter of just disregarding the quality in favour of massive availability