4 ms·

Ooh, thanks for sharing this. I've just gotten around to Kahneman's "Thinking Fast and Slow" (2011) and it seems a LOT of the discussion in the book relies on

Ooh, thanks for sharing this. I've just gotten around to Kahneman's "Thinking Fast and Slow" (2011) and it seems a LOT of the discussion in the book relies on non-reproduced or even reversed results in your list. I hope the book changes significantly in the second half, as the first half is essentially barren after reading your FORRT link.

davetannenbaum 3y ago
Most of Kahneman's own work has held up quite well to replication, and that makes up a large portion of the book. But a lot of the social psych findings he discusses have not fared well to replication attempts.
Zetice 3y ago
Failure to replicate doesn't mean the original findings aren't true or useful. A thousand things besides "the original finding is wrong" can happen to cause a failure of replication. If you limit your understanding to science that is conclusive, you will paralyze yourself. This is the reality of scientific study; there are exceedingly few conclusions, only more questions, confusions, misunderstandings, etc. When scientists say we don't "know" anything, this is what they're talking about. Nothing replicates, everything seems more confusing and less clear the deeper you dig, and the further you stray from the "hard" sciences that are easy to experiment, the more true this gets. Psychology, economics, etc. are insanely complex and effectively impossible to test the way we think of a physics or chemistry experiment.
- huitzitziltzin 3y ago
  On the contrary, failure to replicate should definitely reduce your posterior probability over the original findings being either true OR useful. True findings should replicate. There are complexities - maybe you can't perfectly recreate the original conditions, especially in non-experimental settings - but broadly speaking replication is the right standard to use.
  - Zetice 3y ago
    The presumption here is that it was a true replication, and that's pretty damn hard to accomplish in the soft sciences. It matters, just not nearly as much or as conclusively as in something like physics.
    - davisoneee 3y ago
      If a result is incredibly hard to replicate, that's a good indicator that the effect is either so niche, or so small, that it's not actually telling you any details of the world worth talking about. If 'priming' only works every wet Tuesday in February, after someone has had a cheeseburger with no sauce and took their Grandma to the bingo, perhaps there's some _actual_ interesting artefact explaining it...or it could just be random noise.
      - Zetice 3y ago
        A lot of the studies involved do replicate, just with reduced effects. But the presumption here is that the same experiment is even being run a second time, and my point is that many studies who have the stated goal of replication do so in ways that I find hard to justify. So even when a study claims to "fail to replicate", I don't think we should just take that on face value.
- chmod600 3y ago
  “This is the reality of scientific study; there are exceedingly few conclusions, only more questions, confusions, misunderstandings, etc.“ That goes too far. Some science really does follow the scientific method and documents the methodology well-enough to be replicated. That’s the basis for giving deference to scientists on some topics — they aren’t just experts, they have subjected their ideas to the cold hard scientific method and others have replicated the results. Somewhere along the line we demoted science to just be another expertise, where you can just walk around in a lab coat and have social status and people aren’t supposed to question you. Throw in some jargon. No, that’s not science. They can’t have it both ways. If scientists want deference on important topics, they need to follow the scientific method and not extrapolate too far from established results. No excuses like being “paralyzed” by requiring replicable results.

davetannenbaum 3y ago

Most of Kahneman's own work has held up quite well to replication, and that makes up a large portion of the book. But a lot of the social psych findings he discusses have not fared well to replication attempts.

Zetice 3y ago

Failure to replicate doesn't mean the original findings aren't true or useful. A thousand things besides "the original finding is wrong" can happen to cause a failure of replication. If you limit your understanding to science that is conclusive, you will paralyze yourself. This is the reality of scientific study; there are exceedingly few conclusions, only more questions, confusions, misunderstandings, etc. When scientists say we don't "know" anything, this is what they're talking about. Nothing replicates, everything seems more confusing and less clear the deeper you dig, and the further you stray from the "hard" sciences that are easy to experiment, the more true this gets. Psychology, economics, etc. are insanely complex and effectively impossible to test the way we think of a physics or chemistry experiment.

huitzitziltzin 3y ago
On the contrary, failure to replicate should definitely reduce your posterior probability over the original findings being either true OR useful. True findings should replicate. There are complexities - maybe you can't perfectly recreate the original conditions, especially in non-experimental settings - but broadly speaking replication is the right standard to use.
- Zetice 3y ago
  The presumption here is that it was a true replication, and that's pretty damn hard to accomplish in the soft sciences. It matters, just not nearly as much or as conclusively as in something like physics.
  - davisoneee 3y ago
    If a result is incredibly hard to replicate, that's a good indicator that the effect is either so niche, or so small, that it's not actually telling you any details of the world worth talking about. If 'priming' only works every wet Tuesday in February, after someone has had a cheeseburger with no sauce and took their Grandma to the bingo, perhaps there's some _actual_ interesting artefact explaining it...or it could just be random noise.
    - Zetice 3y ago
      A lot of the studies involved do replicate, just with reduced effects. But the presumption here is that the same experiment is even being run a second time, and my point is that many studies who have the stated goal of replication do so in ways that I find hard to justify. So even when a study claims to "fail to replicate", I don't think we should just take that on face value.
chmod600 3y ago
“This is the reality of scientific study; there are exceedingly few conclusions, only more questions, confusions, misunderstandings, etc.“ That goes too far. Some science really does follow the scientific method and documents the methodology well-enough to be replicated. That’s the basis for giving deference to scientists on some topics — they aren’t just experts, they have subjected their ideas to the cold hard scientific method and others have replicated the results. Somewhere along the line we demoted science to just be another expertise, where you can just walk around in a lab coat and have social status and people aren’t supposed to question you. Throw in some jargon. No, that’s not science. They can’t have it both ways. If scientists want deference on important topics, they need to follow the scientific method and not extrapolate too far from established results. No excuses like being “paralyzed” by requiring replicable results.