6 ms·
Famous cognitive psychology experiments that failed to replicate
- delichon 1y agoApproximate replication rates in psychology: social 37% cognitive 42% personality 55% clinical 44% So a list of famous psychology experiments that do replicate may be shorter. https://www.nature.com/articles/nature.2015.18248 https://www.nature.com/articles/nature.2015.18248
- NewJazz 1y agoI think one would wish the famous ones to be more often replicable.
- sunscream89 1y agoThere may be minute details like having a confident frame of reference for the confidence tests. Cultures, even psychologies might swing certain ideas and their compulsions.
- tomjakubowski 1y agoNonreplicable publications are cited more than replicable ones (2021) > We use publicly available data to show that published papers in top psychology, economics, and general interest journals that fail to replicate are cited more than those that replicate. This difference in citation does not change after the publication of the failure to replicate. Only 12% of postreplication citations of nonreplicable findings acknowledge the replication failure. https://www.science.org/doi/10.1126/sciadv.abd1705 https://www.science.org/doi/10.1126/sciadv.abd1705 Press release: https://rady.ucsd.edu/why/news/2021/05-21-a-new-replication-crisis-research-that-is-less-likely-to-be-true-is-cited-more.html https://rady.ucsd.edu/why/news/2021/05-21-a-new-replication-...
- nitwit005 1y agoThis feels like some sort of truth telling paradox, where if you assume the study is true, then seeing a citation like this means it's likely not true.
- esperent 1y agoThis is at least partially a failure in publication. Once a paper is published, it's usually left up in the same state forever. If it fails to replicate, that data is published somewhere else. So when someone references the paper, and the diligent reader follows up and reads the reference, it looks convincing, just as it did when first published. It's not reasonable to expect the reader, or even the writer, to be so well versed in all the thousands and thousands of papers published that they know when something has failed to be replicated. What we need is for every paper to be published alongside a stats card that is kept up to date. How many times it's been cited, how many times people tried to replicate it, and how many times they failed.
- dlcarrier 1y agoIsn't the unexpected more famous than the expected?
- t_mann 1y agoThanks for providing the reference, that's useful context. Those are awful replication rates, worse than a coin flip. Sounds like the OP can add their own introduction to their list. From the introduction: > Most results in the field do actually replicate and are robust[citation needed]
- glial 1y agoThe incentive of all psychology researchers is to do new work rather than replications. Because of this, publicly-funded psychology PhDs should be required to perform study replication as part of their training. Protocol + results should be put in a database.
- gwd 1y agoHow interesting would it be if every PhD thesis had to have a "replication" section, where they tried to replicate some famous paper's results.
- analog31 1y agoSure, dump it on the lowest level employee, who has the least training and the most to lose. Punish them for someone else's bad research. Grad school already takes too long, pays too little, and involves too much risk of not finishing. And it doesn't solve the problem of people having to generate copious quantities of research in order to sustain their careers. Disclosure: Physics PhD.
- SpaceManNabs 1y agoOne thing that confuses me is that some of these papers were successfully replicated, so juxtaposing them to the ones that have not been replicated at all given the title of the page feels a bit off. Not sure if fair. The ego depletion effect seems intuitively surprising to me. Science is often unintuitive. I do know that it is easier to make forward-thinking decisions when I am not tired so I dont know.
- taeric 1y agoThe idea isn't that it is easier to do things when not tired. It is that you specifically get tired exercising self control. I think that can be subtly confused by people thinking you can't get better at self control with practice? That is, I would think a deliberate practice of doing more and more self control every day should build up your ability to do more self control. And it would be easy to think that that means you have a stamina for self control that depletes in the same way that aerobic fitness can work. But, those don't necessarily follow each other.
- ceckGrad 1y ago>some of these papers were successfully replicated, so juxtaposing them to the ones that have not been replicated at all given the title of the page feels a bit off. Not sure if fair. I don't like Giancotti's claims. He wrote: >This post is a compact reference list of the most (in)famous cognitive science results that failed to replicate and should, for the time being, be considered false. I don't agree with Giancotti's epistemological claims but today I will not bloviate at length about the epistemology of science. I will try to be brief. If I understand Marco Giancotti correctly, one particular point is that Giancotti seems to be saying that Hagger et al. have impressively debunked Baumeister et al. The ego depletion "debunking" is not really what I would call a refutation. It says, "Results from the current multilab registered replication of the ego-depletion effect provide evidence that, if there is any effect, it is close to zero. ... Although the current analysis provides robust evidence that questions the strength of the ego-depletion effect and its replicability, it may be premature to reject the ego-depletion effect altogether based on these data alone." Maybe Baumeister's protocol was fundamentally flawed, but the counter-argument from Hagger et al. does not convince me. I wasn't thrilled with Baumeister's claims when they came out, but now I am somehow even less thrilled with the claims of Hagger et al., and I absolutely don't trust Giancotti's assessment. I could believe that Hagger executed Baumeister's protocol correctly, but I can't believe Giancotti has a grasp of what scientific claims "should" be "believed."
- Terr_ 1y ago> Source: Hagger et (63!) al. 2016 I can't help chuckling at the idea that over 1.98 * 10^87 people were involved in the paper.
- dlcarrier 1y agoIf you were to meet a "normal" person, would you interpret that as meaning "perpendicular" or as meaning "the kind of person that doesn't look at everything like it's a mathematical expression"?
- Terr_ 1y agoNormal: "The kind of person who doesn't go out of their way to put-down other people on HN for being nerdy when a linked article contains a weird editorial interjection that bears an unusual resemblance to a math expression."
- recursive 1y agoUm, actually, the interpretation here is "factorial", not "perpendicular".
- fsckboy 1y agofamous cognitive psychology experiments that do replicate: IQ tests http://www.psychpage.com/learning/library/intell/mainstream.html http://www.psychpage.com/learning/library/intell/mainstream.... in fact, the foundational statistical models considered the gold standard for statistics today were developed for this testing.
- alphazard 1y ago> in fact, the foundational statistical models considered the gold standard for statistics today were developed for this testing. The normal distribution predates the general factor model of IQ by hundreds of years.[0] You can try other distributions yourself, it's going to be hard to find one that better fits the existing IQ data than the normal (bell curve) distribution. [0] https://en.wikipedia.org/wiki/Normal_distribution#History https://en.wikipedia.org/wiki/Normal_distribution#History
- fsckboy 1y agoDarwin's cousin, Francis Galton, for whom the log-normal distribution is often called the Galton distribution, was among the first to investigate psychometrics. not realizing he was hundreds of years late to the game, he still went ahead and coined the term "median" more tidbits here https://en.wikipedia.org/wiki/Francis_Galton#Statistical_innovation_and_psychological_theory https://en.wikipedia.org/wiki/Francis_Galton#Statistical_inn...
- astrange 1y agoSurvivorship bias. You can easily make someone's IQ test not replicate. (Hit them on the head really hard.)
- dlcarrier 1y agoI took an IQ test as a high school student, and one of the subtests involved placing a stack of shuffled pictures in chronological order. I had one series in the incorrect order, because I had no understanding of the typical behavior of snowfall. The test proctor said almost everyone she tested mixed that one up, because it doesn't snow in the area where I live. I have no doubt that IQ tests reproducibly measure the test takers ability to pass tests, as well as to perform in a society that the tests are based on. I think it's disingenuous to attribute IQ to intelligence as a whole though, and it is better understood as an indicator of cultural intelligence. I would expect that, for cultures who's members score below average on IQ tests from the US, an equivalent IQ test created within that culture would show average members of that culture scoring higher than average members of US culture.
- ausbah 1y agoi wonder the replication rate is for ML papers
- PaulHoule 1y agoFrom working in industry and rubbing shoulders with CS people who prioritize writing papers over writing working software I’m sure that in a high fraction of papers people didn’t implement the algorithm they thought they implemented.
- avdelazeri 1y agoDon't get me started, I have seem repos that I'm fairly sure never ran in their presented form. A guy in our lab thinks authors purposefully mess up their code when publishing on GitHub to make it harder to replicate. I'm starting to come around on his theory.
- KingMob 1y agoAnd most medical studies. It's just as bad as social psych, if not worse, because there's real money at stake in churning out new drugs.
- intalentive 1y agoNowadays everyone publishes their code. There’s typically a project page on github.io, a paper on arxiv.org, and a public repo.
- aeve890 1y ago>Source: Stern, Gerlach, & Penke (2020) Wow, what are the odds? https://en.wikipedia.org/wiki/Stern%E2%80%93Gerlach_experiment https://en.wikipedia.org/wiki/Stern%E2%80%93Gerlach_experime...
- NooneAtAll3 1y agoI'm still amazed that wikipedia doesn't have redirect away from its mobile site
- dang 1y ago(It's on my list to rewrite those URLs in HN comments at least)
- NoMoreNicksLeft 1y agoPlease, please, please... can you rewrite reddit links to old.reddit.com too? Not that there's much reason to link there, but it makes my eyes bleed.
- dang 1y agoIt depends if the bulk of the HN community supports it. As you probably know, we already do that for submission URLs.
- dlcarrier 1y agoI thought you were pointing out some bias by comparing the research to previous research from the same authors. It took me far too long to realize that the experiment was from 100 years ago, and you were pointing out that the names were coincidentally the same.
- jbentley1 1y agoThis is a great list for people who want to smugly say "Um, actually" a lot in conversation. Based on my brief stint doing data work in psychology research, amongst many other problems they are AWFUL at stats. And it isn't a skill issue as much as a cultural one. They teach it wrong and have a "well, everybody else does it" attitude towards p-hacking and other statistical malpractice.
- Waterluvian 1y agoUm, actually I’d say it is the responsibility of all scientists, both professional and amateur, to point out falsehoods when they’re uttered, and not an act of smugness.
- rolph 1y ago[um], has contexts but is usually a cue, that an unexpected, off the average, something is about to be said. [actually], is a neutral declaration that some cognitive structure was presented, but is at odds with physically observable fact that will now be laid out to you.
- sputr 1y agoAs someone who's part of a startup (hrpotentials.com) trying to bring truly scientifically valid psychological testing into HR processes .... yeah. We've been at it for almost 7 years, and we're finally at a point where we can say we have something that actually makes scientific sense - and we're not inventing anything new, just commercializing the science! It only took an electrical engineer (not me) with a strong grasp of statistics working for years with a competent professor of psychology to separate the wheat from the chaff. There's some good science there it's just ... not used much.
- PaulHoule 1y agoYeah, this is an era which is notorious for pseudoscience.
- obviouslynotme 1y ago
- hn_throw_250915 1y agoI thought we knew that these were vehicles by wannabe self-help authors to puff up their status for money. See for example “Grit” and “Deep Work” and other bullshit entries in a breathlessly hyped up genre of pseudoscience.
- systemstops 1y agoIs anyone tracking how much damage to society bad social science has done? I imagine it's quite a bit.
- feoren 1y agoWe rack up quite a lot of awfulness with eugenics, phrenology, the "science" that influenced Stalin's disastrous agriculture policies in the early USSR, overpopulation scares leading to China's one-child policy, etc. Although one could argue these were back-justifications for the awfulness that people wanted to do anyway.
- systemstops 1y agoThose things were not done by awful people though - they all thought they were serving the public good. We only judge it as awful now because of the results. Nearly of these ideas (Lysenkoism I think was always fringe) were embraced by the educated elites of the time.
- feoren 1y agoLysenkoism! That's the one. Thank you for reminding me of the name (and for knowing what I was grasping at). I think some "bad people" used eugenics and phrenology to justify prior hate, but they were also effective tools at convincing otherwise "good people" to join them.
- daoboy 1y agoYou are absolutely right. Another interesting example: The man who invented the lobotomy won a Nobel Prize for it.
- izabera 1y agoi'm struggling to imagine many negative effects on society caused by the specific papers in this list
- systemstops 1y ago
- juujian 1y agoNow I want to know which cognitive psychology experiments were successfully replicated though.
- blindriver 1y agoPapers should not be accepted until an independent lab has replicated the results. It’s pretty simple but people are incentivized to not care if it’s replicable because they need the paper to publish to advance their career
- jay_kyburz 1y agoAgreed, and the independent lab should be chosen by the publisher, and be kept secret until results are in.
- mcswell 1y agoIn many cases--longitudinal studies are an example, but not the only one--that's not feasible. And it's often expensive--who would pay for it?
- Animats 1y ago> Most results in the field do actually replicate and are robust [citation needed], so it would be a pity to lose confidence in the whole field just because of a few bad apples. Is there a good list of results that do consistently replicate?
- gwd 1y ago> Smile to Feel Better Effect > Claimed result: Holding a pen in your teeth (forcing a smile-like expression) makes you rate cartoons as funnier compared to holding a pen with your lips (preventing smiling). More broadly, facial expressions can influence emotional experiences: "fake it till you make it." I read this about a decade ago, and started, when going into a situation where I wanted to have a natural smile, grimacing maniacally like I had a pencil in my teeth. The thing is, it's just so silly, it always makes me laugh at myself, at which point I have a genuine smile. I always doubted whether the claimed connection was real, but it's been a useful tool anyway.
- sunscream89 1y agoYeah, the marshmallow one taught me to have patience and look for the long returns on investments of personal effort. I think there may be something to a few of these, and more may need considering regarding how these are conducted. Let’s leave open our credulities for the inquest of time.
- deleted 1y ago[deleted]
- bogtog 1y agoLittle of this is considered cognitive psychology. The vast majority would be viewed as "social psychology" Setting that aside, among any scientific field I'm aware of, psychology has taken the replication crisis most seriously. Rigor across all areas of psychology is steadily increasing: https://journals.sagepub.com/doi/full/10.1177/25152459251323480 https://journals.sagepub.com/doi/full/10.1177/25152459251323...
- WesolyKubeczek 1y ago> Claimed result: Listening to Mozart temporarily makes you smarter. This belongs in a dungeon crawl game. You find an artifact that plays music to you. Depending on the music played (depends on the artifact's enchantment and blessed status), it can buff or debuff your intelligence by several points temporarily.
- epolanski 1y ago[flagged]
- myhf 1y agoCircular reasoning can be used to "prove" anything, so it's not helpful as a basis for policy making.
- us-merul 1y ago> We know for a fact that sex or ethnicity impacts body yet we seem unable to cope with the idea that there are also differences in how brains work. Here is your error. You’re assuming that a physical difference in morphology is linked to behavioral or neural correlates. That’s not the case, since observed statistical- or group-level differences need not be driven by biology. You’re assuming biological determinism, and the evidence for direct genetic effects on behavior isn’t there.
- epolanski 1y agoIt's not an error unless you're able to demonstrate the opposite. I have yet to see studies that demonstrate that different sexes, hormones or even ethnicities do not impact cognitive abilities or higher proficiency in different fields. Whereas I've seen plenty that show that women, on average, demonstrate higher cognitive abilities linked to verbal proficiency, text comprehension or executive tasks. Women also tend to have better memory than men. Facts are that there are genetic differences in how our brains work. And let's not ignore the huge importance of hormones, extremely potent regulators of how we function. To ignore that we have differences that, at large, help explain statistics is asinine.
- us-merul 1y agoAnd how are you able to rule out that societal or environmental effects are the primary driver? How is your argument not circular, that observed differences are therefore the result of biology?
- 1y ago
- picardo 1y agoWell, at least the growth mindset study is not fully debunked yet. It's basically a modern interpretation of what we've known to be true about self-fulfilling prophecies. If you tell children they are can be smart and competent if they work hard, then they will work hard and become smart and competent. This should be a given.
- sunrunner 1y agoNo mention of the Stanford Prison Experiment I notice.
- dlcarrier 1y agoYou'd think it's so far in the past that it isn't even considered, but Zimbardo was elected president of the American Psychological Association in 2002, which wasn't all that long ago.
- runarberg 1y agoAnd during which time APA was complicit in and participated in torturing prisoners at Guantanamo Bay. https://www.apa.org/about/policy/chapter-4b https://www.apa.org/about/policy/chapter-4b
- cindyllm 1y ago[dead]
- insane_dreamer 1y agoIf the "failed replication" was a single study, as in many cases listed here, there is still an open question as to whether the 1) replication study was underpowered (the ones I looked at had pretty small n's), or 2) the re-implementation of the original study was flawed. So I'm not so sure we can quickly label the original studies as "debunked", no more than we can express a high level of confidence in the original studies. (This isn't a comment on any of the individual studies listed.)
- Aurornis 1y ago> Claimed result: Adopting expansive body postures for 2 minutes (like standing with hands on hips or arms raised) increases testosterone, decreases cortisol, and makes people feel more powerful and take more risks. A heuristic I use that is unreasonably good at identifying grifters and charlatans: Unnecessarily invoking cortisol or other hormones when discussing behavioral topics. Influencers, podcasters, and pseudoscience practitioners love to invoke cortisol, testosterone, inflammation, and other generic concepts to make their ideas sound more scientific. Instead of saying "stress levels" they say "cortisol". They also try to suggest that cortisol is bad and you always want it lower, which isn't true. Dopamine is another favorite of the grifters. Whenever someone starts talking about raising dopamine or doing something to increase dopamine, they're almost always being misleading or just outright lying. Health and fitness podcasters are the worst at this right now.
- deleted 1y ago[deleted]
- thecrims0nchin 1y agoI have a draft of a blog post on this. Originally I was going to write about how cortisol isn't always bad, or good, it's just a chemical in us. But then I started noticing the pattern you point out here where I'm not sure anyone uses the cortisol argument in good faith. Everyone who brings up cortisol is usually trying to sell you something
- tryauuum 1y agoclaimed result: Women are more attracted to hot guys during high-fertility days of their cycles wait why not? I hoped I'm attractive at least some days of the month :(
- dlcarrier 1y agoDisturbing fact: The Stanford prison experiment, run by Philip Zimbardo, wasn't reproducible but that didn't stop Zimbardo from using it to promote his ideologies about the impossibility of rehabilitating criminals, or from becoming the president of the American Psychological Association. The APA has a really good style guide, but I don't trust them for actual psychology.
- runarberg 1y agoYes, the APA certainly has a lot to answer for in their history. The Guantanamo Prison torture scandal is still fresh in my memory. https://www.democracynow.org/2007/8/20/apa_members_hold_fiery_town_hall https://www.democracynow.org/2007/8/20/apa_members_hold_fier...
- chatmasta 1y agoHas anyone tried to reproduce it? Good luck convincing an ethics review board to let you try that again. Meanwhile, it’s been reproduced “in vitro” in numerous episodes of atrocity, e.g. Abu Ghraib…
- dlcarrier 1y agoA reality TV show tried, because they have more practical ethics requirements than academic institutions, and were able to run the study as long as everyone agreed to be there and there was no sign of abuse: https://en.wikipedia.org/wiki/The_Experiment https://en.wikipedia.org/wiki/The_Experiment Also, how do you italicize text in your comment?
- chatmasta 1y ago> Also, how do you italicize text in your comment? HN will italicize any string between a pair of asterisks. [0] > practical ethics requirements Practical ethics requirements :) [0] https://news.ycombinator.com/formatdoc https://news.ycombinator.com/formatdoc
- lutusp 1y agoA key factor behind psychology's low replication rate is the absence of theories that define the field. In most science fields, an initial finding can be compared to theory before publication, which may weed out unlikely results in advance. But psychology doesn't have this option -- no theories, so no Litmus test. It's important to say that a psychology study can be scientific in one sense -- say, rigorous and disciplined, but at the same time be unscientific, in the sense that it doesn't test a falsifiable, defining psychological theory -- because there aren't any of those. Or, to put it more simply, scientific fields require falsifiable theories about some aspect of nature, and the mind is not part of nature. Future neuroscience might fix this, but don't hold your breath for that outcome. I suspect we'll have AGI in artificial brains before we have testable, falsifiable neuroscience theories about our natural brains.
- runarberg 1y agoNote that nearly non of these studies are pure cognitive psychology. Most have intersections with social psychology (and I would deem primarily social psychology) or developmental psychology. For example the debunked study on social priming was published in Journal of Personality and Social Psychology. This title would be much more accurate if the author omitted “cognitive” from the title.
- somewholeother 1y agoThe economics of pyschology are the psychology of economics. If you won't trust the process, you will gain no real outcome. What we recieve from the process is not necessarily tangible, but instead a fresh perspective on what may be possible. Thus, the inversion is complete, and we may then move forward.
- seec 1y agoAll the "hypothesis" or supposed "results" are so bonkers than it's an insult to intelligence itself that such things can be "proved" with psych "experiment". Not that it matters, most of the psychology field is inherently bullshit, those are just the example of cases they went so far in the insult to intelligence, no amount of "studies" and rhetoric can save them.
- eviks 1y agoGiven how long the whole field has been malicious/incompetent failing at basic statistics and sweeping it under the rug, I think that rather then discarding the experiments that don't replicate the better baseline is to discard everything and wait for the future generation of better cognitive psychologists that come up with any good discovery that is widely replicated?
- eska 1y agoI recently read the lifework book of a nobel prize winning psychologist and it was full of these disproven experiments. As a non-psychologist my trust in the experts is extremely low.
- patrickhogan1 1y agoBefore dunking on psychology for not replicating, remember this is a cross-discipline problem. In biomedicine, Amgen could reproduce only 6/53 “landmark” preclinical cancer papers and Bayer reported widespread failures.
- camgunz 1y agoDear lazyweb: is there the opposite of this list anywhere?
- HK-NC 1y agoIIRC the 2013 "racism predicted by telling leading questions" one and ita predecessor, which is listed here but also says there is slight trend toward replication, is just based on implicit association tasks. So you have a green and red button for good and bad, and then a word pops up and you have less a second to choose which button to press. Oversimplifying complex thought processes in my opinion is junk psychology.
- chatmasta 1y agoIt’s also self-referential because there is no objective measure of “racism,” so how can you even measure whether someone is “more racist” based on reaction time to stereotypical stimuli? “No objective measure” pretty much sums up the whole field, to be honest. I started on a CS & Psych double major, did about eight psych courses, and then decided it was mostly a joke once I got to the quantitative portions. But those courses were very useful for general life skills. Developmental psychology in particular was packed with dense lessons about how we learn as children… social psych was a good overview of all the “well-known” experiments… etc.
- djoldman 1y agoI'm surprised that dunning Kruger isn't listed.
- Ferret7446 1y agoDevil's advocate, is it possible that humans have psychologically changed since the original experiments?