6 ms·
There is a severe demographic problem with this study (among others). A current trend in the social sciences is using Amazon Mechanical Turk and Prolific to run
by somenameforme 4y ago
There is a severe demographic problem with this study (among others). A current trend in the social sciences is using Amazon Mechanical Turk and Prolific to run surveys. These sites are based around paying people pennies to compete menial tasks (like complete surveys), and populated with large numbers of "professional" menial task executers. This study offered the big bucks of $1.50 - $2.25. There are two major problems with this:
1) It doesn't matter how you normalize a sample when that sample is a niche subsample of society to begin with. This has always been a problem with the social sciences which previously relied heavily on undergraduates in social science classes, but that is going to at least be much closer to something like a representative sample than Amazon Mechanical Turk or Prolific can offer.
2) The people carrying out the tasks are motivated primarily by blazing through as many studies as possible while doing the bare minimum to get paid, and many fail at even that. In this study ctrl+f for "attention check." To this study's credit, they at least included numerous attention checks, but it emphasizes the issue. On average about 1 out of 6 participants were excluded for failing to correctly answer questions along the lines of "Please select C as the answer to this question."
Journals are likely indulging this because it's going to generate endless "impact" (which is increasingly becoming a euphemism for clickbait), but it's undermining the entire field more than ever before.
- manimino 4y agoWhy would undergraduates be more representative of society than Mechanical Turk users? Surely the Turk users span a much wider range of age and geolocation.
- bowsamic 4y agoIt always amazes me how such studies are simply left up to the academics, because clearly taking representative samples of the population will need some kind of help from the government to organise it
- incone123 4y agoProbably just requires a bigger budget than most academics have. I got paid £50 to spend half an hour giving feedback on a new website, in a 1:1 conversation with a researcher. Granted the funding was from government, but Kantar did the organisation side.
- deleted 4y ago[deleted]
- MichaelCollins 4y agoI think it's not possible to do ethically or practically, because only some sorts of people will willingly subject themselves to the testing, examination and prodding of of social sciences, even when bribed. And if you tried to force it on them using a government, which would be unethical, you still have the practical problem of the data you get from people under duress being garbage.
- marcosdumay 4y agoThere are many companies that specialize on this. Usually, it takes picking random people at many different streets, but does not require government intervention.
- origin_path 4y agoNo, all you need to do is hire a professional polling company. They also have biases but they're far closer to providing representative samples than MT is. Academia doesn't do this because it's expensive and they have a cultural bias towards quantity over quality.
- montebicyclelo 4y agoI saw another study on hn [1] that "showed that people found accountants and data analysts boring". However, this was all based on forms filled in by workers on MTurk and another crowd sourcing company... [1] https://news.ycombinator.com/item?id=30764152 https://news.ycombinator.com/item?id=30764152
- qikInNdOutReply 4y agoSorry, but the public social sciences are dead anyway. If you want to do research on humanity you go were the actual data is. FGN. Everything else is peanuts.
- danuker 4y agoWhat is FGN?
- verteu 4y agoI think it's "Facebook/Google/Netflix."
- danuker 4y agoThat makes sense. Big Tech in general has data on billions.
- qikInNdOutReply 4y agoNSA. Assume you have the conversation of billions. Its pretty easy to design "Virtual experiments", that just play out or have played out as life plays out. Select from Everyone //Filter the Environment with constraint to avoid polution where(Person.isAlcoholic and Person.Age.inRange(12,49)) //Define experimental condition experimentOutcome(State.Change(Person.isAlcoholic) upon Person.Graph.Depth(1).Death) Gets you all acoholics quitting after death of a nearby relative. That way, you get almost pure data and archieve what neuro sciences and psychology could never dream off. To really know people better then they even ever knew themselves. So sad, as this is "steering" technology, we will never get to see it. We do not even get to contemplate its existence.
- imgabe 4y agoI wonder if you could get good survey data by disguising it as a Buzzfeed quiz on Facebook. Just make the title something like "Which Kardashian are you?", then ask whatever questions you want and randomly select a Kardashian at the end.
- martijnvds 4y ago"You are Gul Dukat."
- Der_Einzige 4y agoCardassian, Kardashian, all the same to me!
- erybodyknows 4y ago‘Keeping up with the Cardassians’
- Arrath 4y agoI would seriously question my life choices.
- dTal 4y agoI suspect that's exactly what most of those "quizzes" actually are.
- vermote 4y agoI believe that's what Cambridge Analytica did.
- comfypotato 4y agoAs someone who has never taken, and will probably never take, a Buzzfeed quiz: I think it probably risks the same kind of sampling bias. Not sure what the best way to overcome this is. Joke: government mandated surveys with monetary punishments based on your net worth if you fail to take them.
- 4y ago
- exo-pla-net 4y agoThe need for representative sampling is something understood by first year undergraduates on first exposure to the concept. What you've written is the reddit equivalent of criticizing a study with "correlation does not imply causation". Yeah, scientists are familiar with the idea, believe it or not. Scientists in fact go to great lengths to ensure their sampling is representative, and studies published in good journals are carefully scrutinized for non-representative sampling. If you can't conceive of how a person working at Mechanical Turk is overwhelmingly likely to just be a normal person, and not your weird conception of them, I'd encourage you to read Thinking Fast and Slow. Contempt for scientists is not a good look.
- rwmj 4y agoI think his point is that it's impossible to normalise such a sample. For example if I sampled only toddlers about their attitudes to ice cream then there's no way to extrapolate from that sample to the general population, and it's the same thing for "people who are willing to fill in a survey for $2".
- MichaelCollins 4y agoUnderstanding the need for representative samples is not the same as actually getting representative samples. It's not a problem of scientists being ignorant or lazy or even underfunded. They know what they need, and they simply cannot get it. They try to make do anyway, but the quality of their work still suffers because of these factors which are out of their control. Not all fields of inquiry are equally tractable to the scientific method, so they should not be treated with equal reverence.
- eliasmacpherson 4y agoWithout getting into a snark, that is not a particularly good book to recommend. https://slate.com/technology/2016/12/kahneman-and-tversky-researched-the-science-of-error-and-still-made-errors.html https://slate.com/technology/2016/12/kahneman-and-tversky-re... "Taken all together, the chapters Schimmack looked at earned an average grade of C-minus."
- 4y ago
- neilwilson 4y agoThere's also the perennial problem that polls are not public opinion. The biases within polls are legion and difficult to eliminate in any society with privacy provisions.
- red_admiral 4y agoIt's at least a small step up from when many studies of human behaviour were done on psychology freshmen.
- psychstudio 4y agoI wrote about this problem a few years ago. Even more relevant today: https://www.psychstudio.com/articles/research-participation https://www.psychstudio.com/articles/research-participation
- ryangittins 4y agoMaybe if American society valued science the same we we value justice, we could compel people to occasionally participate in surveys and studies the way we compel them to participate in jury duty. We see the latter as a fact of life and occasional annoyance for a cause greater than ourselves. Maybe we should treat a contribution to science the same way. Many countries require some military service of young people—we could ask people to contribute in all sorts of ways.
- rr888 4y agoMaybe if social science was real science instead of quackery people would want to help.
- gpm 4y agoIf we didn't already have mandatory jury duty, I don't think there's a hope in hell that you could get a bill passed requiring it today. If you did manage to get a bill passed, I think the courts would almost certainly strike it down. We have become much less keen on requiring people to work against their will. Jury duty is grandfathered in, but new things of that kind wouldn't be lightly accepted.
- ryangittins 4y agoThis is true of a lot of things. Imagine if libraries didn't exist and someone got the bright idea for one today. It would never fly.
- origin_path 4y agoIn a world of open source and Wikipedia, that's a bold claim.
- ryangittins 4y agoThose aren't funded by taxpayers like libraries are.
- 4y ago