6 ms·
Classic and contemporary psychology experiments replicated
- sometime 8y agoIs there a list of the experiments and the effect sizes and how they differed from the original studies?
- bluetwo 8y agoYes, in the paper, you can to follow the links.
- ppeetteerr 8y agoTLDR; only 50% succeeded replication. You don't have to feel so bad about not reading that Malcolm Gladwell book, now.
- bluetwo 8y agoI'm just going to point out that the ridiculous idea that using a hard to read font improves retention of information was disproven.
- andrelaszlo 8y agoShort summary: - The 28 findings was repeated in more than 60 laboratories all over the world resulting in a median sample size of 7,157 - Overall, 14 of the 28 findings (50%) replicated successfully - The replication team obtained the original materials to ensure faithful replications of the original findings - All 28 studies underwent formal peer review at the journal prior to conducting the studies - All of the studies were preregistered at OSF - Researchers can predict whether findings will replicate or not in surveys and prediction markets with pretty good accuracy. (prediction market) Full paper (100 pages): https://psyarxiv.com/9654g/ https://psyarxiv.com/9654g/
- andrelaszlo 8y agoList of studies, and the outcome: 1 Direction & SES (Huang et al., 2014) p<.001 2 Structure and goal-pursuit (Kay et al., 2014) p=0.97 3 Incidental disfluency (Alter et al., 2007) p=0.67 4 Moral Foundations (Graham et al., 2009) p<.001 5 Affect and Risk (Rottenstreich & Hsee, 2001) p=0.77 6 Priming consumerism (Bauer et al., 2012) p=0.15 7 Correspondence Bias (Miyamoto & Kitayama, 2002) p<.001 8 Disgust & Homophobia (Inbar et al., 2009) p=0.56 9 Incidental Anchors (Critcher & Gilovich, 2008) p=0.25 10 Social Value Orientation (Van Lange et al., 1997) p<.001 11 Trolley Dilemma 1 (Hauser et al., 2007) p<.001 12 SMS & Well-Being (Anderson et al., 2012) p=0.58 13 False Consensus 1 (Ross et al., 1977) p=0.23 14 False Consensus 2 (Ross et al., 1977) p<.001 15 Position & Power (Giessner & Schubert, 2007) p=0.31 16 Framing (Tversky & Kahneman, 1981) p=0.43 17 Trolley Dilemma 2 (Hauser et al., 2007) p=0.42 18 Tempting Fate (Risen & Gilovich, 2008) p=0.01 19 Actions are Choices (Savani et al. 2010) p<.001 20 Intuitive Reasoning (Norenzayan et al. 2002) p<.001 21 Less is Better (Hsee, 1998) p<.001 22 Moral Typecasting (Gray & Wegner, 2009) p<.001 23 Moral Cleansing (Zhong & Liljenquist, 2006) p=0.08 24 Assimilation and Contrast (Schwarz et al., 1991) p=0.39 25 Choosing or Rejecting (Shafir, 1993) p=0.10 26 Priming warmth (Zaval et al., 2014) p=0.01 27 Intentional Side Effects (Knobe, 2003) p<.001 28 Direction and Similarity (Tversky & Gati, 1978) p=0.99
- andrelaszlo 8y agoLet me know if I messed anything up and I'll fix it, it was a quick copy-paste.
- jonnycomputer 8y agowhich journals would be nice. quick way of getting idea of how important these papers actually were.
- sys_64738 8y agoNone of the classics.
- conjectures 8y agoI'm not much of a psychologist, but I can see at least one paper by Khaneman, who won the economics Nobel for that sort of stuff. So not sure if you're being sarcastic or not?
- iandanforth 8y ago"It was surprising that even with these efforts we were only able to obtain support for the original findings for half of the studies. These results do not definitively mean that the original findings were wrong, but they do suggest that they are not as robust as might have been assumed." - Michelangelo Vianello, one of the project leads. Yes, yes they do. It's ok to say the original findings were wrong. It's ok to re-assert the null hypothesis. The unwillingness to do so undermines the process and is disconcerting to say the least.
- tlb 8y agoFailure to replicate doesn't prove the original findings were wrong. Other things that can cause replication failure are: - Poor explanation of experimental procedure, so the replication doesn't match the original experiment. - Awareness of the effect may invalidate it. For example, it'd be hard to replicate the Stanford Prison Experiment because every psychology undergrad has heard the story. - Change in social conditions or prevailing attitudes. Consumer behavior (the subject of some of these studies) has changed significantly over the decades. Homophobia (another study) varies dramatically over time and place.
- iandanforth 8y agoWe should continue this argument over coffee, hope you're doing well!
- skrebbel 8y agoPlease record the audio, i'm curious who convinces who. (or just continue here on HN of course)
- pfortuny 8y agoNot exactly: it is ok to say that there were no original findings. This is a tiny semantic detail but important: if the author does not support his findings, it is OK to re-assert the null hypothesis but not to say that "the findings were wrong". What I mean is: "no experiment" does not mean the same as "wrong experiment".
- jonnycomputer 8y agoI read psych papers. Regularly. I don't know any of these aside from Tversky's. Psychology is a big field. I just find it hard to accept that these papers are representiative of much of anything.
- tvanantwerp 8y agoLooking through the paper, even the half that replicated generally seem to have done so with greatly diminished magnitude.
- tylerjwilk00 8y agoThe thing with psychology experiments is that the fundamental thing being tested is not immutable. This makes it different than many other science experiments where concrete parts of reality are tested. Psychology really only tests the minds of humans in a global society at a point in time. If future minds change enough the results may no longer be valid.
- deleted 8y ago[deleted]
- jhedwards 8y agoYour statement in itself is a hypothesis to be proved. There are psychology experiments designed specifically to ascertain what parts of the mind are hard-wired and which are depended on nurture/culture. I am not a Psychologist but I can give one example: Piaget hypothesized that the mind developed by first accumulating low-level sensory/motor data, and then progressively building higher levels of abstraction on that foundation. If that is true, then that process itself is not related to society or culture and should be observable so long as the biology underpinning cognitive development doesn't radically change.
- tylerjwilk00 8y agoThe mind is mutable. Psychology is different than other "hard" sciences. I only say "results _may_ no longer be valid" and I'd say this study proves that. If you go deep enough into the mind to the point where social environment no longer influences the behavior you may be getting closer to biology than psychology but point taken.
- pbhjpbhj 8y ago>This makes it different than many other science experiments where concrete parts of reality are tested. // Maybe, or maybe immutability is axiomatic in those other areas.
- _pastel 8y agoFun game: read through the summary of each experiment (psyarxiv.com/9654g) and try to guess whether it replicated or not before they tell you. I got 20/28. Some of them were obvious (#3 - no, using a bad font does not make people answer questions more carefully; #22 - yes, people attribute more responsibility to adults than to babies). But there were some big surprises for me: #6 - Participants read a hypothetical water conservation dilemma, then rate how much they trust the people in the scenario to conserve water. Sometimes the people are called "consumers" and sometimes "individuals". Does it make a difference? Apparently yes. When they're called "consumers", you expect them to conserve significantly less water than when they're "individuals". #24 - if you ask "How satisfied are you with your relationship?" and then "How satisfied are you with your life-as-a-whole?", are the results more highly correlated than if you ask those questions in the opposite order? I thought for sure they would be - relationship happiness is one of several contributing factors to your overall happiness, so if it's fresh on your mind, you'll weigh it more heavily when you answer the life-as-a-whole question. Apparently not.
- c3534l 8y agoI'm about halfway through doing just that and I think a pattern has emerged: does there exists a plausible, concrete (you could put it into words) mechanism for cause and effect? That is, can a person tell you why the finding exists. For instance, a semantic association between capitalism and greed does not provide a plausible mechanism for why mentioning capitalism should make one greedy. The gay one failed that test, but otherwise that seems to be what's going on. A lot of these papers aren't testing a scientific theory. They're testing vague, half-baked ideas and associations.
- dorchadas 8y agoI'd love for someone to go through and do this with education research. It's an absolute joke, and I'd argue it's worse than psychology research. And it doesn't help teachers are making students refuse to use old literature (or maybe I'm just pissed I can't quote older studies in my 'literature review' for my 'research paper'...)
- hnmullany 8y agoHere are some of the details of some of the unreplicated studies (not scientifically specific - but you can always read the paper) Replication Failures - Disgust sensitivity predicts homophobia (no it doesn’t) - Assimilation and contrast - the order of asking specific vs. general questions about the same domain matters (no it doesn’t) - Priming Consumerism - being told you’re acting as a consumer reduces your trust in others (no it doesn’t) - Moral Cleansing - people exposed to unethical things have an urge to cleanse themselves physically (no they don't) - Priming Warmth - on warm days people have higher belief in global warming (no they don’t) - Structure & Goal Pursuit - being exposed to structure improves goal pursuit (no it doesn't) - Incidental Disfluency - making tasks incidentally difficult (hard to read fonts) - improves analytic performance (no it doesn’t)
- throw2016 8y agoPoor choice of headline for the original article given 'replicated' indicates success. This was an intensive attempt to replicate but 14 of the 28 did not replicate. That is a 50 percent failure rate inspite of using over 186 researchers in 60 labs worldwide. The widely reported 2012 study could not replicate nearly 100 of 100 studies. And in this case if this kind of comprehensive effort can't replicate half the studies it raises serious questions about the original studies and adds more fuel to the reproducibility crisis. Given studies are routinely used to justify and implement policies [1], and taken seriously as points of authority it's not just academic. There needs to be much larger discussion on the use of studies to direct policy that impact real lives, and the level of oversight, additional replication and other credibility building measures that need to be considered. [1] https://www.sciencedirect.com/science/article/pii/S0016328717300472 https://www.sciencedirect.com/science/article/pii/S001632871...