5 ms·

A Bot Panic Hits Amazon Mechanical Turk

20 points by yarapavan 8y ago

yarapavan 8y ago
The blog post mentioned in the article - https://www.maxhuibai.com/blog/evidence-that-responses-from-repeating-gps-are-random https://www.maxhuibai.com/blog/evidence-that-responses-from-... In the past day or two, I discovered that a large number of responses in my latest Mturk survey appear to be random responses. I detect that a large portion of these random responses have repeating GPS locations. So far, a relatively large number of social psychological researchers also seem to have noticed a drop of data quality in their Mturk data and detected concerning patterns using the GPS method (see the related discussion threads on Facebook PsychMAP).
herogreen 8y ago
I find this article interesting because I did not know that MTurk was used by psychologist to survey people. The only usage I had heard of was data labeling for machine learning. This surveying usage seems very strange because I expect the data to be very biased: the (human) respondents will be located in places where the average salary is close to what is paid by MTurk, will be abe to use a computer, etc...
- snapspans 8y ago
  If you're interested, there is more on that (linked from the article) here: http://journal.sjdm.org/10/10630a/jdm10630a.html http://journal.sjdm.org/10/10630a/jdm10630a.html Effectively, without the bots, the paper contends that the Turkers are at least as representative a sample as undergraduate students.
  - sireat 8y ago
    One thing that the paper does not address is MTurkers becoming professional survey takers who recognize typical psychology surveys thus skewing the responses. Personal anecdata: I took many MTurk surveys for about a month in 2012. It is mind crushing work. There were countless near identical surveys presumably by psychology undergrads. Pretty much anything in Kahneman's Thinking Fast and Slow was there. Very quickly you recognize the typical priming experiment and then is your response any valid? If you know the idea behind the baseball bat and ball pricing question then is your correct answer made in a split second helping? Most survey givers wanted MTurkers with high ratings. If you think about it, those MTurkers with high rating would have likely seen all those survey questions already. Thus experienced MTurkers would really distort the answers compared to a random sample of undergraduates. Maybe you can compensate for that somehow.
georgeecollins 8y ago
Isn't it ironic that a service named after a person pretending to be a machine may be subverted by machines pretending to be people?

20 points by yarapavan 8y ago

The blog post mentioned in the article - https://www.maxhuibai.com/blog/evidence-that-responses-from-repeating-gps-are-random https://www.maxhuibai.com/blog/evidence-that-responses-from-... In the past day or two, I discovered that a large number of responses in my latest Mturk survey appear to be random responses. I detect that a large portion of these random responses have repeating GPS locations. So far, a relatively large number of social psychological researchers also seem to have noticed a drop of data quality in their Mturk data and detected concerning patterns using the GPS method (see the related discussion threads on Facebook PsychMAP).

herogreen 8y ago

I find this article interesting because I did not know that MTurk was used by psychologist to survey people. The only usage I had heard of was data labeling for machine learning. This surveying usage seems very strange because I expect the data to be very biased: the (human) respondents will be located in places where the average salary is close to what is paid by MTurk, will be abe to use a computer, etc...

snapspans 8y ago
If you're interested, there is more on that (linked from the article) here: http://journal.sjdm.org/10/10630a/jdm10630a.html http://journal.sjdm.org/10/10630a/jdm10630a.html Effectively, without the bots, the paper contends that the Turkers are at least as representative a sample as undergraduate students.
- sireat 8y ago
  One thing that the paper does not address is MTurkers becoming professional survey takers who recognize typical psychology surveys thus skewing the responses. Personal anecdata: I took many MTurk surveys for about a month in 2012. It is mind crushing work. There were countless near identical surveys presumably by psychology undergrads. Pretty much anything in Kahneman's Thinking Fast and Slow was there. Very quickly you recognize the typical priming experiment and then is your response any valid? If you know the idea behind the baseball bat and ball pricing question then is your correct answer made in a split second helping? Most survey givers wanted MTurkers with high ratings. If you think about it, those MTurkers with high rating would have likely seen all those survey questions already. Thus experienced MTurkers would really distort the answers compared to a random sample of undergraduates. Maybe you can compensate for that somehow.

georgeecollins 8y ago

Isn't it ironic that a service named after a person pretending to be a machine may be subverted by machines pretending to be people?