5 ms·

Mechanical Turk: The New Face of Behavioral Science?

ramidarigaz 12y ago
I'm doing a Master's thesis in the intersection between Cognitive Science and Machine Learning. The group I work with looooves Mechanical Turk. It's such a cheap way to do small validation studies.
- tectonic 12y ago
  I've used it for data gathering, but found it to be a pretty homogenous group of people. What has your experience been?
  - sireat 12y ago
    There are various clusters of homogeneous people. You have your classic /r/beermoney students, same as you would get from regular studies on campus. Then you get your bored housewives of US, and oversees workers mostly Indians/Philipinos. Once you realize these large groups of people exist, you can still get useful work done by accounting for obvious biases. Disclaimer: I've used MTurk both for surveys as a requester and also used it as a Turker for a month two years ago, whenever I got bored. I saw so many Kahneman inspired studies it was not even funny at the end. (Monty Hall problems, priming, $1.10 bat+ball problem, etc etc) . I ended up making around $6 an hour. The best problems were porno classification problems. I suppose that is standard stuff you do in psychology these days.
  - ramidarigaz 12y ago
    I haven't actually looked at the demographic information we've collected. Most of the work the group does has to do with memory and engagement, which varies enough person-to-person that their demographic doesn't matter so much.
john_horton 12y ago
If you're looking for a discussion of running economics-style experiments on MTurk (as well as some advice on keeping online experiments internally valid): http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1591202 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1591202
runeks 12y ago
I think Amazon's Mechanical Turk (MT) is fascinating. I really wish Amazon would release more figures on its usage, rather than only the number of registered users[1]. It would be really interesting to see how many people earn their wages through this service. The chart in the article says 16% in Feb '10 [2], but I think it would be really interesting to see 1) how much money they earn, 2) how many hours they work, 3) where they live, 4) how much Amazon pays out in total in MT wages (all anonymized data, of course). [1] > The only numbers that we share regarding our Worker population are these two -- which reflect our current state: Over 500K registered Workers from over 190 countries worldwide. https://forums.aws.amazon.com/thread.jspa?threadID=58891 https://forums.aws.amazon.com/thread.jspa?threadID=58891 [2] http://pix-media.s3.amazonaws.com/blog/835/ScreenShot2014-10-14at4.58.57PM.png http://pix-media.s3.amazonaws.com/blog/835/ScreenShot2014-10...
- mandalar12 12y ago
  Here is an article [1] trying to estimate that kind of numbers (wage, number of active workers, etc.) from various sources and questionning the ethics of using MTurk for science purposes. The bottom line is: one third of tasks are achieved by people relying on MTurk (as primary or secondary money input) having working conditions that would not be considered acceptable in any first-world country (hourly wages around $2, working long hours with, obviously, no kind of worker advantage or protection). [1] https://hal.archives-ouvertes.fr/file/index/docid/569450/filename/LastWords_AMT.pdf https://hal.archives-ouvertes.fr/file/index/docid/569450/fil...
  - jbarrios 12y ago
    Harvard recently published a paper on the mturk demographics, you can find it here: http://scholar.harvard.edu/files/dtingley/files/whoarethesepeople.pdf http://scholar.harvard.edu/files/dtingley/files/whoarethesep... If you're looking for something less academic, try out this great blog: http://experimentalturk.wordpress.com/ http://experimentalturk.wordpress.com/
jbarrios 12y ago
Crowdsourcing marketplaces offer unique advantages to researchers that are simply impossible to achieve with traditional methodologies, here's a few: 1) External Validity: Today, almost all research labs in Universities have to source their participants from their respective undergraduate population. Studies suggest that the results gathered from 'convenience samples' cannot be generalized to the general population. That is, the results from experiments conducted on 19-23 year old Stanford undergraduates may not be representative of the general population (http://www.mccombs.utexas.edu/research/~/media//Files/MSB/Research/Recent-Publications/2014Q1/PetersonR http://www.mccombs.utexas.edu/research/~/media//Files/MSB/Re...). MTurk offers researchers a more diverse subject pool in age, geography, language, and culture. (https://ccrma.stanford.edu/~jieun5/publications/2012-icmpc-mturk.pdf https://ccrma.stanford.edu/~jieun5/publications/2012-icmpc-m...) 2) Cross-cultural and International Studies: Conducting cross-cultural research though traditional methodologies requires dealing with differing national requirements for protecting human subjects;variation in data collection processes in primary care practices; data transmission among participants; duties and tariffs on necessary instruments; fluctuation in currency exchange rates; incapacitation of coinvestigators; complex administration of funds; financing the additional, legitimate costs of collaboration; sustaining strong personal relationships among coinvestigators; and accepting longer time frames than would otherwise be expected. (http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1466730/ http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1466730/). The diversity on Mechanical Turk facilitates cross-cultural and international research at a very low cost (labs.yahoo.com/files/mturkmethods.pdf) 3) Faster Theory/Experiment Loop: One implicit goal in research is to maximize the efficiency with which one can go from generating hypotheses to testing them, analyzing the results, and updating the theory. Ideally, the limiting factor in this process is the time it takes to do careful science, but all too often, research is delayed because of the time it takes to recruit subjects and recover from errors in the methodology. With access to a large pool of subjects online, recruitment is vastly simplified (labs.yahoo.com/files/mturkmethods.pdf) Traditional researchers have raised questions about the quality associated with the data collected from participants. The empirical evidence strongly suggests that the quality of data collected from mturk participants meets or exceeds the standards associated with published research (http://www.pomona.edu/Academics/departments/psychology/files/Buhrmester%20-Crowdsourcing-Amazon-MTurk.pdf http://www.pomona.edu/Academics/departments/psychology/files...) (http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0057410 http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjourna...). Evidence that Mechanical Turk is a valid means of collecting data is consistent and continues to accumulate (Mason & Suri, 2012) Disclaimer: I'm a crowdsourcing technology advisor to the department of Psychology at Vanderbilt University, VizCogLab, and UVic Center for Autism Research. I'm also the co-founder and CEO of Cognilab Technologies, an online platform that allows researchers to publish their experiments on mturk.

I'm doing a Master's thesis in the intersection between Cognitive Science and Machine Learning. The group I work with looooves Mechanical Turk. It's such a cheap way to do small validation studies.

tectonic 12y ago
I've used it for data gathering, but found it to be a pretty homogenous group of people. What has your experience been?
- sireat 12y ago
  There are various clusters of homogeneous people. You have your classic /r/beermoney students, same as you would get from regular studies on campus. Then you get your bored housewives of US, and oversees workers mostly Indians/Philipinos. Once you realize these large groups of people exist, you can still get useful work done by accounting for obvious biases. Disclaimer: I've used MTurk both for surveys as a requester and also used it as a Turker for a month two years ago, whenever I got bored. I saw so many Kahneman inspired studies it was not even funny at the end. (Monty Hall problems, priming, $1.10 bat+ball problem, etc etc) . I ended up making around $6 an hour. The best problems were porno classification problems. I suppose that is standard stuff you do in psychology these days.
- ramidarigaz 12y ago
  I haven't actually looked at the demographic information we've collected. Most of the work the group does has to do with memory and engagement, which varies enough person-to-person that their demographic doesn't matter so much.

john_horton 12y ago

If you're looking for a discussion of running economics-style experiments on MTurk (as well as some advice on keeping online experiments internally valid): http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1591202 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1591202

runeks 12y ago

I think Amazon's Mechanical Turk (MT) is fascinating. I really wish Amazon would release more figures on its usage, rather than only the number of registered users[1]. It would be really interesting to see how many people earn their wages through this service. The chart in the article says 16% in Feb '10 [2], but I think it would be really interesting to see 1) how much money they earn, 2) how many hours they work, 3) where they live, 4) how much Amazon pays out in total in MT wages (all anonymized data, of course). [1] > The only numbers that we share regarding our Worker population are these two -- which reflect our current state: Over 500K registered Workers from over 190 countries worldwide. https://forums.aws.amazon.com/thread.jspa?threadID=58891 https://forums.aws.amazon.com/thread.jspa?threadID=58891 [2] http://pix-media.s3.amazonaws.com/blog/835/ScreenShot2014-10-14at4.58.57PM.png http://pix-media.s3.amazonaws.com/blog/835/ScreenShot2014-10...

mandalar12 12y ago
Here is an article [1] trying to estimate that kind of numbers (wage, number of active workers, etc.) from various sources and questionning the ethics of using MTurk for science purposes. The bottom line is: one third of tasks are achieved by people relying on MTurk (as primary or secondary money input) having working conditions that would not be considered acceptable in any first-world country (hourly wages around $2, working long hours with, obviously, no kind of worker advantage or protection). [1] https://hal.archives-ouvertes.fr/file/index/docid/569450/filename/LastWords_AMT.pdf https://hal.archives-ouvertes.fr/file/index/docid/569450/fil...
- jbarrios 12y ago
  Harvard recently published a paper on the mturk demographics, you can find it here: http://scholar.harvard.edu/files/dtingley/files/whoarethesepeople.pdf http://scholar.harvard.edu/files/dtingley/files/whoarethesep... If you're looking for something less academic, try out this great blog: http://experimentalturk.wordpress.com/ http://experimentalturk.wordpress.com/

jbarrios 12y ago

Crowdsourcing marketplaces offer unique advantages to researchers that are simply impossible to achieve with traditional methodologies, here's a few: 1) External Validity: Today, almost all research labs in Universities have to source their participants from their respective undergraduate population. Studies suggest that the results gathered from 'convenience samples' cannot be generalized to the general population. That is, the results from experiments conducted on 19-23 year old Stanford undergraduates may not be representative of the general population (http://www.mccombs.utexas.edu/research/~/media//Files/MSB/Research/Recent-Publications/2014Q1/PetersonR http://www.mccombs.utexas.edu/research/~/media//Files/MSB/Re...). MTurk offers researchers a more diverse subject pool in age, geography, language, and culture. (https://ccrma.stanford.edu/~jieun5/publications/2012-icmpc-mturk.pdf https://ccrma.stanford.edu/~jieun5/publications/2012-icmpc-m...) 2) Cross-cultural and International Studies: Conducting cross-cultural research though traditional methodologies requires dealing with differing national requirements for protecting human subjects;variation in data collection processes in primary care practices; data transmission among participants; duties and tariffs on necessary instruments; fluctuation in currency exchange rates; incapacitation of coinvestigators; complex administration of funds; financing the additional, legitimate costs of collaboration; sustaining strong personal relationships among coinvestigators; and accepting longer time frames than would otherwise be expected. (http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1466730/ http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1466730/). The diversity on Mechanical Turk facilitates cross-cultural and international research at a very low cost (labs.yahoo.com/files/mturkmethods.pdf) 3) Faster Theory/Experiment Loop: One implicit goal in research is to maximize the efficiency with which one can go from generating hypotheses to testing them, analyzing the results, and updating the theory. Ideally, the limiting factor in this process is the time it takes to do careful science, but all too often, research is delayed because of the time it takes to recruit subjects and recover from errors in the methodology. With access to a large pool of subjects online, recruitment is vastly simplified (labs.yahoo.com/files/mturkmethods.pdf) Traditional researchers have raised questions about the quality associated with the data collected from participants. The empirical evidence strongly suggests that the quality of data collected from mturk participants meets or exceeds the standards associated with published research (http://www.pomona.edu/Academics/departments/psychology/files/Buhrmester%20-Crowdsourcing-Amazon-MTurk.pdf http://www.pomona.edu/Academics/departments/psychology/files...) (http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0057410 http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjourna...). Evidence that Mechanical Turk is a valid means of collecting data is consistent and continues to accumulate (Mason & Suri, 2012) Disclaimer: I'm a crowdsourcing technology advisor to the department of Psychology at Vanderbilt University, VizCogLab, and UVic Center for Autism Research. I'm also the co-founder and CEO of Cognilab Technologies, an online platform that allows researchers to publish their experiments on mturk.