10 ms·
Daniel Kahneman calls for ‘daisy chain’ of psychology replications
- dfc 14y agoThis is a meta-piece about another article. The actual article: http://www.nature.com/news/nobel-laureate-challenges-psychologists-to-clean-up-their-act-1.11535 http://www.nature.com/news/nobel-laureate-challenges-psychol... The actual article is worth a read, I was a little intimidated after reading "here’s some overmatter that didn’t make it into the piece because of length" but the actual article is only 712 words. The over-matter is 500 words. More background about the problem: "Replication studies: Bad copy" http://www.nature.com/news/replication-studies-bad-copy-1.10634 http://www.nature.com/news/replication-studies-bad-copy-1.10...
- bambax 14y ago"Replication studies: Bad copy" is very interesting. Sample quote: “I've seen students spending their entire PhD period trying to replicate a phenomenon, failing, and quitting academia because they had nothing to show for their time.” We say that a study "succeeds" when it illustrates a novel behavior and "fails" when the expected behavior doesn't happen. This is a very slippery use of words. A finding that can't be replicated isn't a finding. It's a rumor, an urban myth, but not a scientific step forward. A study that "fails to replicate" a previous finding isn't a "failure" -- it succeeded at debunking a false belief! And it should be granted wide access to publishing. (Same thing happens in medical studies BTW, where it's arguably much more dangerous).
- chii 14y ago> A study that "fails to replicate" a previous finding isn't a "failure". Yep. the problem is that to the "laymen", it seems a bunch of people wasted time trying to re-do what someone else has "already" done, and failed. Whether the failure was due to incompetance, chance, or an actual failure to replicate (meaning it is a significant), to the mind of the laymen, they are one and the same. this then end up being difficult for people who might fund such studies to fund these sorts of studies (despite it being quite useful), and leads to people seeking "newer" things to research. its a bit of a chilling effect really.
- StavrosK 14y agoWho are these laymen, and why are they reviewing papers?
- chii 14y agothey are not reviewing papers, nor are they the fund contributors. They are the general population, whose sentiments affect the way society sees things overall. For example, a lot of people are now rightfully scared of nuclear tech (due to recent disasters). And yet, most people dont understand that neclear energy is quite safe (compared to coal, say). If a political leader decides to invest in nuclear energy, they instantly become unpopular, and thus lies the problem of funding via such channels.
- StavrosK 14y agoSure, but they aren't the ones deciding what gets published or not. The study that tried to replicate another study but failed has already been done, and it's not up to the laymen to decide whether or not to publish the results. Nor can we predict which studies will fail to duplicate results and not fund them, otherwise we wouldn't have to make them. Results of failure are still results, and they represent valuable knowledge. Too bad the people who got them consider them a failure.
- lars512 14y agoIf there's an obvious flaw in the existing study, you can make something out of a new study which doesn't contain that flaw. But it's very hard to convince yourself (and others) that replicating/failing to replicate someone's work is the same level of contribution as the original work. What would be great: a centralised place for people to share their attempts to replicate a work. If several groups have failed, and nobody has succeeded, then there's certainly room for a rigorous, joint debunking of the experiment. If, as is often the case, there's some subtlety to get the experiment right, then the discussion around it between groups actually enhances the original work.
- aba_sababa 14y agoWe act like psychology was ever a legitimate science in the first place...
- rayiner 14y agoCognitive psychology is quite legitimate. You should read some of Kahneman work. He's one of the pioneers in putting both psychology and economics on a firm empirical footing.
- aggie 14y agoPsychology is a broad field with many subfields. Some areas are harder sciences than others (psychobiology/physiology, sensation and perception, neuroscience, some cognitive science, etc.), and some are admittedly very hand-wavy (emotion, personality, just about anything Freud said, etc.), but to dismiss all of psychological science as illegitimate is to ignore a tremendous amount of progress in understanding and human well-being that has come in the past century.
- pgeorgi 14y ago"hand-wavy ... just about anything Freud said" - There's the issue: In pop science and public perception, psychology is nearly 100% Freud (but luckily, this is changing). Public perception doesn't matter much for research in the field, but it doesn't help credibility.
- eavc 14y ago"emotion, personality, just about anything Freud said, etc." Emotion and personality are valid subjects of scientific study as well. There has been a lot of unscientific opinion on those subjects (Freud being a good example), but that doesn't mean there isn't good science done there as well.
- lutusp 14y ago> Emotion and personality are valid subjects of scientific study as well. Only if experimental results can be obtained using objective scientific methods, with control groups and strict experimental protocols, and only if the results can be successfully replicated. In other words, they cannot be studied scientifically. That's the meaning of the linked article: no replication. The article doesn't say such studies can be replicated, it discusses how to deal with the fact that they cannot.
- SparrowOS 14y agoGod says... consulters Eat departed inflicted eating Latins tie God says... two ends thereof. 25:20 And the cherubims shall stretch forth their wings on high, covering the mercy seat with their wings, and their faces shall look one to another; toward the mercy seat shall the faces of the cherubims be. 25:21 And thou shalt put the mercy seat above upon the ark; and in the ark thou shalt put the testimony that I shall give thee. 25:22 And there I will meet with thee, and I will commune with thee from above the mercy seat, from between the two cherubims which are upon the ark of the testimony, of all things which I will give thee in commandment unto the children of Israel.
- FrojoS 14y agoRichard P. Feynman on pseudo science: http://www.youtube.com/watch?v=IaO69CF5mbY http://www.youtube.com/watch?v=IaO69CF5mbY BTW, I come from the field of robotics and there, hardly any published result is replicated either.
- overbroad 14y agoIMHO, Bargh should redo some of his old expts himself. If he's so sure they are repeatable, then why not? His stuff is foundational to the entire field.
- icebraining 14y agoAs an ignorant non-scientist, I assumed it'd be better for someone else to replicate the studies, to avoid the same biases; wouldn't it?
- lutusp 14y ago> As an ignorant non-scientist, I assumed it'd be better for someone else to replicate the studies, to avoid the same biases; wouldn't it? Not at all ignorant. It is a given that a replication should be carried out by a disinterested party, one neither an advocate of, nor critical of, the original work.
- overbroad 14y agoYeah, that's usually how it'd done. But from what I've read (not just from the Nature article, I did actually read his papers years ago), this guy basically founded the entire field of what became "priming". It's all on shaky ground now, whether that's justified or not. I'm not sure it would hurt for him to question his own conclusions, in addition to having others repeat some studies. Time to swallow the pride. Studies are supposed to be repeatable. But in practice a lot of research never gets repeated. I'm sure he must feel a little insulted to have his life's work called into question, even if the cause is other investigators who were fraudulent or incompetent, or even if the questions are arising only indirectly because of unrelated research that happens to use priming. IANAS, but psychology is obviously not the hardest of the sciences. I think in other areas like e.g. chemistry and biology, it's more difficult to pass off methods and results that are not repeatable _if_ they are going to be foundational and used for lots of future research by others. If some other lab can't get the right results using your methods, I would think they will make some noise. They might first think they are themselves at fault, but they would probably make contact with others outside their lab and make it known, in some way, that these methods were not working for them. Whereas I can't see that as being as common in psychological research. It's too easy if some method is not working to just "make it work". ;)
- lutusp 14y ago> Daniel Kahneman calls for ‘daisy chain’ of psychology replications A welcome and overdue call to action. But I predict the challenge will be ignored, because the practitioners of this psychological specialty know the effort will fail, and that failure will destroy what remains of their credibility.
- tokenadult 14y agoThere is now an effort to set up a new open-access journal of psychology that will encourage best research practices like sharing data with other researchers as a built-in part of submitting articles, and taking other steps to communicate (and thus allow checking of) the study method and hypothesis. http://chronicle.com/blogs/percolator/building-the-perfect-journal/30794 http://chronicle.com/blogs/percolator/building-the-perfect-j... A local friend (a mathematician who works as a staff researcher on human behavioral genetics research) and I will be in the weekly journal club meeting later today at the University of Minnesota journal club. This week's topic in the journal club is discussion of Uri Simonsohn, the "data detective." Several of my recent submissions to Hacker News have been of readings related to our discussion today. My local friend sent out this background discussion by email yesterday: "On Wednesday, September 26, Econ Nobelist and psychologist Daniel Kahneman sent an open email message to a group of social priming researchers and one of them forwarded the message to Ed Yong, the science journalist who interviewed Uri Simonsohn for one of this week's readings. This is that email message: http://www.nature.com/polopoly_fs/7.6716.1349271308!/suppinfoFile/Kahneman%20Letter.pdf http://www.nature.com/polopoly_fs/7.6716.1349271308!/suppinf... "Ed Yong contacted Kahneman and interviewed him, then he wrote this short article for Nature News which appeared Wednesday, October 3 Nobel laureate challenges psychologists to clean up their act: Social-priming research needs “daisy chain” of replication http://www.nature.com/news/nobel-laureate-challenges-psychologists-to-clean-up-their-act-1.11535 http://www.nature.com/news/nobel-laureate-challenges-psychol... "and this entry in his blog: Nobel laureate Daniel Kahneman calls for ‘daisy chain’ of psychology replications http://blogs.discovermagazine.com/notrocketscience/2012/10/04/daniel-kahneman-daisy-chain-replications-priming-psychology/ http://blogs.discovermagazine.com/notrocketscience/2012/10/0... "The story was also picked up by Chronicle of Higher Education on Thursday night, October 4: Daniel Kahneman Sees ‘Train-Wreck Looming’ for Social Psychology http://chronicle.com/blogs/percolator/daniel-kahneman-sees-train-wreck-looming-for-social-psychology/31338 http://chronicle.com/blogs/percolator/daniel-kahneman-sees-t... "In related news, Uri Simonsohn posted his paper on statistical detection of data fabrication on the Social Science Research Network site on July 22, 2012 (draft dated 7/21). He has since posted a revised draft, dated July 29, which can be downloaded from this page: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2114571 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2114571