6 ms·

This is a really optimistic take. I agree with you, ultimately, this is positive because it shows that it's possible for psychology to be done and have widespre

This is a really optimistic take. I agree with you, ultimately, this is positive because it shows that it's possible for psychology to be done and have widespread meaningful results. In the short term though this seems extremely negative as it indicates that the psychology field and researchers we have today are producing substandard results. Two parts of this article increase my negative feelings. First, they mention that these 28 studies were selected because they were well known and influential. If well known and influential studies only have a 50% reproducibility rate - what about the average study? Another point I find very troubling is that people betting online were able to predict, with high accuracy, which studies were accurate and which weren't. This tells me that the problem is an "Emperor's New Clothes" type of problem and not just a challenge with the fundamental difficulty of the subject since the subject is so yielding to the public's intuition. Instead of a problem with the fundamental difficulty, the problem seems to be with the psychology of the researchers - being unable to diagnose and prevent methodological flaws in their research. This class of failure is much less excusable to me.

lifeisstillgood 8y ago
For me the unrepeatable experiments suggest something even more interesting - that unless the originals were flawed or fraud, then something about the people tested has changed. We are all more aware of conmen movies, of fake news, or have read popsci books and now know what to look out for.
- lifeisstillgood 8y ago
  ok having looked deeper into this it seems some of the i replicated studies are ... counter intuitive.
jonex 8y ago
"If well known and influential studies only have a 50% reproducibility rate - what about the average study?" Actually, they might be better off. In the related article* they mention: "Beyond statistical issues, it strikes me that several of the studies that didn’t replicate have another quality in common: newsworthiness." It could be that many well known works are well known because their finding are counterintuitive, and therefore more interesting. However, this could also imply that the prior probability is lower - hence a p = 0.05 result is more likely to be due to chance than for a study finding something more in line with prior understanding of things. * https://www.theatlantic.com/science/archive/2018/08/scientists-can-collectively-sense-which-psychology-studies-are-weak/568630/ https://www.theatlantic.com/science/archive/2018/08/scientis...
repolfx 8y ago
The article that goes deeper into the betting markets says that they suspect many of the traders were people invested in the reproducibility crisis and related research, that is, they are more knowledgable than the general public would be. That doesn't take away from your point, but the problem here may be more refined than "the public" vs "psychologists". It may be more like "some psychologists" vs "journal editors and other psychologists".
- ALittleLight 8y ago
  What I wrote was influenced by the fact that I had taken a "Psychology Replication Quiz" which I'll link to below. The quiz presents you with the thesis of a study and asks whether you think it will replicate. When I took it the first time I got ten questions correct out of ten. I took the quiz again a moment before writing this comment and got nine out of ten (I think I was hurrying this time). The average person taking the quiz gets seven out ten correct. The depressing part about this quiz is how easy it is. You just have to think for a moment if what they are describing matches what you know about human nature. Some statements just feel very intuitive and likely, and those replicate. Other statements seem quite wild and abstract, and those don't replicate. I haven't seen a list of the 28 papers they looked at here, but I'd bet it is similar. Some of the findings are incredible (and not reproducible) while others are intuitive and reproducible. I'd encourage you to take the quiz below and see if you feel the same way about the results as I do. Psychology Replication Quiz - https://80000hours.org/psychology-replication-quiz/ https://80000hours.org/psychology-replication-quiz/

lifeisstillgood 8y ago

For me the unrepeatable experiments suggest something even more interesting - that unless the originals were flawed or fraud, then something about the people tested has changed. We are all more aware of conmen movies, of fake news, or have read popsci books and now know what to look out for.

lifeisstillgood 8y ago
ok having looked deeper into this it seems some of the i replicated studies are ... counter intuitive.

jonex 8y ago

"If well known and influential studies only have a 50% reproducibility rate - what about the average study?" Actually, they might be better off. In the related article* they mention: "Beyond statistical issues, it strikes me that several of the studies that didn’t replicate have another quality in common: newsworthiness." It could be that many well known works are well known because their finding are counterintuitive, and therefore more interesting. However, this could also imply that the prior probability is lower - hence a p = 0.05 result is more likely to be due to chance than for a study finding something more in line with prior understanding of things. * https://www.theatlantic.com/science/archive/2018/08/scientists-can-collectively-sense-which-psychology-studies-are-weak/568630/ https://www.theatlantic.com/science/archive/2018/08/scientis...

repolfx 8y ago

The article that goes deeper into the betting markets says that they suspect many of the traders were people invested in the reproducibility crisis and related research, that is, they are more knowledgable than the general public would be. That doesn't take away from your point, but the problem here may be more refined than "the public" vs "psychologists". It may be more like "some psychologists" vs "journal editors and other psychologists".

ALittleLight 8y ago
What I wrote was influenced by the fact that I had taken a "Psychology Replication Quiz" which I'll link to below. The quiz presents you with the thesis of a study and asks whether you think it will replicate. When I took it the first time I got ten questions correct out of ten. I took the quiz again a moment before writing this comment and got nine out of ten (I think I was hurrying this time). The average person taking the quiz gets seven out ten correct. The depressing part about this quiz is how easy it is. You just have to think for a moment if what they are describing matches what you know about human nature. Some statements just feel very intuitive and likely, and those replicate. Other statements seem quite wild and abstract, and those don't replicate. I haven't seen a list of the 28 papers they looked at here, but I'd bet it is similar. Some of the findings are incredible (and not reproducible) while others are intuitive and reproducible. I'd encourage you to take the quiz below and see if you feel the same way about the results as I do. Psychology Replication Quiz - https://80000hours.org/psychology-replication-quiz/ https://80000hours.org/psychology-replication-quiz/