4 ms·

> Participants (N = 196; obtained via Amazon Mechanical Turk) That's bound to be a representative sample.

by mseebach 11y ago

tinco 11y ago
Why would it not be?
- 78666cdc 11y ago
  It's clearly not a random sample of the population.
  - return0 11y ago
    ... and my guess is if they had started with the standard approach (recruit university students) their results wouldn't replicate.
  - tinco 11y ago
    For most psychological research you don't need a very random sample, nor do you need a very large sample. It's got to do with the fact that our brains largely function the same. Your sample has to be large and random enough to be fairly confident that you didn't accidentally pick a significant amount of people with some sort of mental divergence (i.e. very low IQ, very high IQ, autism, psychopathy, etc..) that could be relevant. I don't think picking nearly 200 people from Amazon Turk is going to risk that. But perhaps someone should research what kind of people are on Amazon Turk. Fun exercise, let's modify the conclusion to reflect the specific population bias we think the paper might have: People working for Amazon Turk are more likely to support conservative ideas when they also are susceptible to bullshit. What is a much more important question: What is the distribution of conservative/non-conservative supporters in the group, and how significant is the measured correlation? edit: Just as a disclaimer, I made this comment based on the psych master thesis presentation I've attended of a friend when I was at university where I posed the same question (N was only 16, and only local students were surveyed). He explained to me that for his particular research (pertaining correlation between auditory senses and motor skills) a small group was sufficient because of the fundamental brain structure we (mostly) all share. Whether that holds up for more complex research like this I don't know, I'm not a psych student.
    - coldtea 11y ago
      >For most psychological research you don't need a very random sample, nor do you need a very large sample. Heck, as long as you get published and get a grant, any size or representativeness of sample will do (even straight made up statistics, if there's no big fear of being found out).
    - raverbashing 11y ago
      Most people using AMT are people with lower/limited incomes, see r/mturk
    - dlss 11y ago
      The view you're espousing is wrong. When you actually test if what you're calling human universals actually are universal, you generally find they're not: "Broad claims about human psychology and behavior based on narrow samples from Western societies are regularly published in leading journals. This review suggests not only that substantial variability in experimental results emerges across populations in basic domains, but that standard subjects are in fact rather unusual compared with the rest of the species - frequent outliers. The domains reviewed include visual perception, fairness, categorization, spatial cognition, memory, moral reasoning and self-concepts. This review (1) indicates caution in addressing questions of human nature based on this thin slice of humanity, and (2) suggests that understanding human psychology will require tapping broader subject pools. We close by proposing ways to address these challenges." (The thin slice of humanity referred to above is westernized college students. Perhaps mechanical turkers do not suffer from this bias? seems unlikely though) http://www2.psych.ubc.ca/~ara/Manuscripts/Weird_People_BBS_Henrichetal.pdf http://www2.psych.ubc.ca/~ara/Manuscripts/Weird_People_BBS_H... also see http://www.scientificamerican.com/podcast/episode/psychology-studies-biased-toward-we-10-08-07/ http://www.scientificamerican.com/podcast/episode/psychology... and http://www.slate.com/articles/health_and_science/science/2013/05/weird_psychology_social_science_researchers_rely_too_much_on_western_college.html http://www.slate.com/articles/health_and_science/science/201... for pop write-ups.
      - tinco 11y ago
        Thanks! That's interesting, seems I might have had it wrong. I should've brought that up during my friends master thesis presentation, maybe he wouldn't have got his diploma ;) Are psychology researchers in general aware of this review? Obviously my friends master thesis was just a small inconsequential study, but if the universals that are now commonly used have turned out to not actually be true wouldn't that mean huge swathes of researchers have to retract/redo their research?
- wbillingsley 11y ago
  Even the article says it isn't. "we want to note that the sample of the present study probably is not representative of the US as our study is restricted to the specific sample of Amazon Mechanical Turk workers and has a relatively small sample size for an online survey. Thus, one cannot make inferences about the entire population of the US (or other populations of other countries)" plosOne is unfortunately fairly gamable with this sort of thing.
Tloewald 11y ago
It's at least as representative as the 40-50 college freshmen that are used for most such studies.

> Participants (N = 196; obtained via Amazon Mechanical Turk) That's bound to be a representative sample.

tinco 11y ago

Why would it not be?

78666cdc 11y ago
It's clearly not a random sample of the population.
- return0 11y ago
  ... and my guess is if they had started with the standard approach (recruit university students) their results wouldn't replicate.
- tinco 11y ago
  For most psychological research you don't need a very random sample, nor do you need a very large sample. It's got to do with the fact that our brains largely function the same. Your sample has to be large and random enough to be fairly confident that you didn't accidentally pick a significant amount of people with some sort of mental divergence (i.e. very low IQ, very high IQ, autism, psychopathy, etc..) that could be relevant. I don't think picking nearly 200 people from Amazon Turk is going to risk that. But perhaps someone should research what kind of people are on Amazon Turk. Fun exercise, let's modify the conclusion to reflect the specific population bias we think the paper might have: People working for Amazon Turk are more likely to support conservative ideas when they also are susceptible to bullshit. What is a much more important question: What is the distribution of conservative/non-conservative supporters in the group, and how significant is the measured correlation? edit: Just as a disclaimer, I made this comment based on the psych master thesis presentation I've attended of a friend when I was at university where I posed the same question (N was only 16, and only local students were surveyed). He explained to me that for his particular research (pertaining correlation between auditory senses and motor skills) a small group was sufficient because of the fundamental brain structure we (mostly) all share. Whether that holds up for more complex research like this I don't know, I'm not a psych student.
  - coldtea 11y ago
    >For most psychological research you don't need a very random sample, nor do you need a very large sample. Heck, as long as you get published and get a grant, any size or representativeness of sample will do (even straight made up statistics, if there's no big fear of being found out).
  - raverbashing 11y ago
    Most people using AMT are people with lower/limited incomes, see r/mturk
  - dlss 11y ago
    The view you're espousing is wrong. When you actually test if what you're calling human universals actually are universal, you generally find they're not: "Broad claims about human psychology and behavior based on narrow samples from Western societies are regularly published in leading journals. This review suggests not only that substantial variability in experimental results emerges across populations in basic domains, but that standard subjects are in fact rather unusual compared with the rest of the species - frequent outliers. The domains reviewed include visual perception, fairness, categorization, spatial cognition, memory, moral reasoning and self-concepts. This review (1) indicates caution in addressing questions of human nature based on this thin slice of humanity, and (2) suggests that understanding human psychology will require tapping broader subject pools. We close by proposing ways to address these challenges." (The thin slice of humanity referred to above is westernized college students. Perhaps mechanical turkers do not suffer from this bias? seems unlikely though) http://www2.psych.ubc.ca/~ara/Manuscripts/Weird_People_BBS_Henrichetal.pdf http://www2.psych.ubc.ca/~ara/Manuscripts/Weird_People_BBS_H... also see http://www.scientificamerican.com/podcast/episode/psychology-studies-biased-toward-we-10-08-07/ http://www.scientificamerican.com/podcast/episode/psychology... and http://www.slate.com/articles/health_and_science/science/2013/05/weird_psychology_social_science_researchers_rely_too_much_on_western_college.html http://www.slate.com/articles/health_and_science/science/201... for pop write-ups.
    - tinco 11y ago
      Thanks! That's interesting, seems I might have had it wrong. I should've brought that up during my friends master thesis presentation, maybe he wouldn't have got his diploma ;) Are psychology researchers in general aware of this review? Obviously my friends master thesis was just a small inconsequential study, but if the universals that are now commonly used have turned out to not actually be true wouldn't that mean huge swathes of researchers have to retract/redo their research?
wbillingsley 11y ago
Even the article says it isn't. "we want to note that the sample of the present study probably is not representative of the US as our study is restricted to the specific sample of Amazon Mechanical Turk workers and has a relatively small sample size for an online survey. Thus, one cannot make inferences about the entire population of the US (or other populations of other countries)" plosOne is unfortunately fairly gamable with this sort of thing.

Tloewald 11y ago

It's at least as representative as the 40-50 college freshmen that are used for most such studies.