9 ms·
It’s worth noting that many of the results in Thinking, Fast and Slow didn’t hold up to replication. It’s still very much worth reading in its own right, but n
by mistercow 3y ago
It’s worth noting that many of the results in Thinking, Fast and Slow didn’t hold up to replication.
It’s still very much worth reading in its own right, but now implicitly comes bundled with a game I like to call “calibrate yourself on the replication crisis”. Playing is simple: every time the book mentions a surprising result, try to guess whether it replicated. Then search online to see if you got it right.
- zug_zug 3y agoYeah I wouldn't read too much into any single study. But what I would defend vigorously is System1 / System2 distinction as something so clear/fundamental that you can see it constantly once you understand it.
- disgruntledphd2 3y agoThat's not him though. Like, it was in all my cog psych textbooks more than twenty years ago, with cites back in the 80s (which weren't him). This is my favourite paper of theirs: http://stats.org.uk/statistical-inference/TverskyKahneman1971.pdf http://stats.org.uk/statistical-inference/TverskyKahneman197... I got into a bunch of trouble with some reviewers of my thesis for referencing this repeatedly.
- gattilorenz 3y agoIt’s also very common in psychology theories, I haven’t read “Thinking, fast and slow” but I imagine there’s more than Kahneman’s own papers cited: https://en.wikipedia.org/wiki/Dual_process_theory https://en.wikipedia.org/wiki/Dual_process_theory
- zug_zug 3y agowow, it looks like "dual process" theory is basically the same thing. I don't know if there's a better text on dual-process theory out there (perhaps by the original authors), but regardless of who originated it, I think it's something worth learning about for everyone (and if you don't have a better source then Thinking Fast and Slow is a very good one).
- rileyphone 3y agoIt's just such a bad name though.
- underdeserver 3y agoIn software we often call it fastpath and slowpath :)
- deleted 3y ago[deleted]
- lupire 3y agoIt's been called "emotion / intuition" and "logic" for centuries or millennia before the goofy System name was invented.
- raincole 3y agoIronically people like System 1/2 more than intuition/logic because the terms sound more like they are coined by System 2.
- derbOac 3y ago... except the distinction was being made in various forms long before Kahneman, and does get questioned. When you start to poke at it, what's intuitive starts to seem less so. https://journals.sagepub.com/doi/full/10.1177/1745691612460685 https://journals.sagepub.com/doi/full/10.1177/17456916124606... (that's a link to a defense of dual process theories, but it makes clear there's increasing criticism of them)
- cjblomqvist 3y agoDoes anyone have a link to a publicly accessible version of this paper?
- derbOac 3y agoI think this should work: https://scottbarrykaufman.com/wp-content/uploads/2014/04/dual-process-theory-Evans_Stanovich_PoPS13.pdf https://scottbarrykaufman.com/wp-content/uploads/2014/04/dua... There's review paper coming from a more critical perspective in Psych Bulletin or Psych Review I was looking for, but I couldn't find it atm.
- paulpauper 3y agoI wonder if it's better to have a lot of small hits or a few big hits and many misses in regard to replication. If the studies which have the greatest implications replicate, then maybe many misses is not that bad.
- btilly 3y agoThat's an interesting theoretical question. Unfortunately the reality is that the more interesting and quotable the result is, the less likely it is to replicate. So replication problems most strongly hit things that seem like they should have the greatest implications. Kind of a "worst of all worlds" scenario.
- ethanbond 3y agoAnd critically, scientific publications are incentivized likewise to publish the most outlandish claims they can possibly get away with. That same incentive affects individual scientists, who disproportionately submit the most outlandish claims they can possibly get away with. The boring stuff -- true or not -- is not worth the hassle of putting into a publishable article.
- btilly 3y agoAnd then the most outlandish of these are picked up by popular science writers. Who proceed to mangle it beyond recognition, and add a random spin. This then goes to the general public. Some believe the resulting garbage. And wind up with weird ideas. Others use that garbage to justify throwing out everything that scientists say. And then double down on random conspiracy theories, denialism, and pseudoscience. I wish there was a solution to this. But everyone is actually following their incentives here. :-(
- lupire 3y agoThe scientists push it on the pop writers, to created a Personal Brand and an industrial complex around their pet theory.
- btilly 3y agoThe density of non-replicable results varies by chapter. You can ignore anything said in chapter 4 about priming for example. See https://replicationindex.com/2020/12/30/a-meta-scientific-perspective-on-thinking-fast-and-slow/ https://replicationindex.com/2020/12/30/a-meta-scientific-pe... for more.
- PheonixPharts 3y agoWhat's wild to me is that anyone could read chapter 4 and not look up the original papers in disbelief. Long before the controversy was public I was reading that book and, despite claims that the reader must believe the findings, it sounded like nonsense to me. So I looked up the original paper to see what the experiment set up was, and it was unquestionably a ridiculous conclusion to draw from a deeply flawed experiment. I still never understood how that chapter got through without anyone else having the same reaction I did.
- dwighttk 3y agoIn those times, that was exactly the kind of thing that people wanted to believe
- foobiekr 3y agoI had exactly this reaction to Malcolm gladwell. It is completely obvious that gladwell across multiple books has never once read one of his references and consistently misrepresents what they say.
- kbenson 3y agoI have a slightly different take on him, which comes to the same ultimate end on how I view his work. As he's shifted from primarily a journalist to primarily a storyteller, he's chosen to sacrifice additional information and accuracy in lieu of telling a consistent and compelling narrative that support what he thinks is the important thing to take away, not necessarily what you would take away were you to review all the same information yourself. Under that understanding, I find him fun to listen to. The things he "reports" on/illuminates are interesting, but at this point I don't assume he's giving them an even handed representation, so his conclusions are not necessarily my own, and at best it's a set of things to look into and research myself if I find my interest piqued after a fun or interesting story is told.
- cranium 3y agoFor what it's worth, Kahnman answered a post that scrutinized the effect of priming: https://replicationindex.com/2017/02/02/reconstruction-of-a-train-wreck-how-priming-research-went-of-the-rails/comment-page-1/#comment-1454 https://replicationindex.com/2017/02/02/reconstruction-of-a-...
- pigscantfly 3y agoThanks for sharing this -- I read the book maybe a decade ago and largely discounted it as non-replicable pop-sci; this changed my opinion of Kahneman's perspective and rigor (for the better!)
- airstrike 3y agoIt looks like it's a bit more nuanced than that. What I saw from the link was some debate about what holds and what doesn't for various forms of "priming"
- coffeebeqn 3y agoThe general idea is very simple. Tactical vs strategic thinking are two different things and it’s good to be aware of that. I don’t know that that needs to be proven or disproven
- EthanHeilman 3y ago19th Century definition of tactics aas being everything that happens within the range of cannons and strategy as everything that happens outside of cannon range, fits well to thinking fast (tactics) and slow (strategy).
- sampo 3y ago> It’s worth noting that many of the results in Thinking, Fast and Slow didn’t hold up to replication. Irony is, Kahneman had himself written a paper warning about generalizing from studies with small sample sizes: "Suppose you have run an experiment on 20 subjects, and have obtained a significant re- sult which confirms your theory (z = 2.23, p < .05, two-tailed). You now have cause to run an additional group of 10 subjects. What do you think the probability is that the results will be significant, by a one-tailed test, separately for this group?" "Apparently, most psychologists have an exaggerated belief in the likelihood of successfully replicating an obtained finding. The sources of such beliefs, and their consequences for the conduct of scientific inquiry, are what this paper is about." Then 40 years later, he fell into the same trap. He became one of the "most psychologists". http://stats.org.uk/statistical-inference/TverskyKahneman1971.pdf http://stats.org.uk/statistical-inference/TverskyKahneman197...
- yawboakye 3y agopsychology isn’t science. it’s a grave mistake to read/interpret it a such. does that mean it’s useless? of course not: some of the findings (and i use findings very loosely) help us adjust our prior probabilities. if we’re right in the end, we were lucky. otherwise we just weren’t.
- andrelaszlo 3y agoThat's an unexpected position for me. How do you define science? Could it be a science, according to you, or is there something fundamentally non-scientific about it?
- yawboakye 3y agoit’s fundamentally unscientific at this point. much of our current science lies in the realm of natural law. so far we haven’t found any laws that govern human behavior. what we know, with considerable certainty, is that behavior can be positively influenced. but at the point of action, nothing we know of compels any specific/predictable behavior. until we have found rigid laws of reasons that apply to both the brute and the civilized, any ‘discoveries’ of psychology are reports of someone’s idiosyncrasies, imho.
- knightoffaith 3y agoIs it not scientific to say that X property is true of human behavior more often that it is not with statistical significance?
- sn9 3y agoThere's nothing about the scientific method that requires the process to output tidy little laws to be deserving of being called science. Some fields are quite lucky that the universe is so elegantly organized, but for that isn't true for the overwhelming majority of fields with as many degrees of freedom as biological systems and anything more complex. That doesn't mean we can't conduct experiments that reproduce.
- worik 3y ago
- davetannenbaum 3y agoHis own work held up very well to replication. It's when he is citing the work of other scholars (in particular, that of social psychologists) that doesn't hold up well to replication.
- canjobear 3y ago“When I describe priming studies to audiences, the reaction is often disbelief . . . The idea you should focus on, however, is that disbelief is not an option. The results are not made up, nor are they statistical flukes. You have no choice but to accept that the major conclusions of these studies are true.”
- importantbrian 3y agoThis game is doubly valuable when the surprising result confirms one of your existing beliefs. I'm pretty good about doing this for surprising results that contradict a belief I hold, but I have to be extra disciplined about doing it when it confirms one of my beliefs.
- acchow 3y ago> calibrate yourself on the replication crisis I imagine that in 30 years, it will become clear that individual humans display enormous diversity, their diversity increasing as societal norms relax, and their behavior changing as the culture around them change. As such, replication is hopeless and trying to turn "psychology" into a science was a futile endeavor. That is not to say that psychology cannot be helpful, just that we cannot infer rational conclusions or predictions from it the same way we can from hard sciences. Self help books are enormously helpful, but they're definitely not science either.
- Systemling0815 3y ago[dead]
- arpa 3y agoyou have no idea what "psychology" is, do you?
- tome 3y agoI have a similar point of view as acchow, so it's possible I don't understand psychology either. Could you enlighten us about what it actually is?
- arpa 3y agoit is a positivistic science, i.e. it studies observable phenomena using the scientific method. The days of Freud and Jung where you could just smoke cigars and fart out ideas about collective unconsciousness and anal fixation are long over. Experiments are conducted, confounding variables controlled, and hypotheses (including H0) are tested. Granted, it's not as easy as in physics where you just drop a ball repeatedly and note down the results, buf it doesn't in any way make it a "softer" science. To equate psychology with self-help books is akin to equating LLMs with Markov chains.
- richrichie 3y agoExperiments involving grad students dont correlate well with how (normal) people behave in real life.
- darby_eight 3y agoI imagine this is just as fun to play with unsurprising results.
- wslh 3y ago> It’s worth noting that many of the results in Thinking, Fast and Slow didn’t hold up to replication. Beyond this specific issue, are psychology experiments and issues time and culture sensitive? I think so [1] [1] https://www.ssoar.info/ssoar/bitstream/handle/document/42104/ssoar-2010-henrich_et_al-The_weirdest_people_in_the.pdf?sequence=1 https://www.ssoar.info/ssoar/bitstream/handle/document/42104...
- faeriechangling 3y ago“Many” is hyperbole, “Some” is more fair and his results stood up or the replication crisis better than most of his contemporaries.
- yreg 3y agoThis is unfortunately the case for many books on human behavior. Sure, Dan Ariely comes to mind, but the field itself is very tricky. I don't think we - people used to STEM - appreciate how difficult behavioral psychology is. In STEM, we are used to isolating experiments down, so there are as few variables as possible. And we are used to well-designed experiments being reproducible if everyone does what they are supposed to do right. In the study of human behavior there are always countless uncontrollable variables, every human is a bit different and it is very difficult to discover something that would apply generally. Also, pretty much all of the research is done on western population of European descent. This is why I take all behavioral claims with a large grain of salt, but I still have respect for the researchers doing their best in the field.
- abdulhaq 3y agoI don't agree, I think if you understand science in general then you realize at an early age e.g. 20 that social/behavioral science is at best a pseudoscience
- jongjong 3y agoIt's so weird that the stuff about priming was supposedly debunked, yet if you look around at what happened to society over the past few years, I've been blown away by how suggestible and controllable people are.
- kirso 3y agoI believe same was the case for "Growth mindset"
- pkkm 3y ago> many of the results My impression is that the priming chapter is bunk, but the rest has generally held up. Is that no longer true?