4 ms·
It's positive news mostly - they took 15,000 volunteers and ran 28 "big" studies across them, working with original teams to get the nuances right. So they had
by lifeisstillgood 8y ago
It's positive news mostly - they took 15,000 volunteers and ran 28 "big" studies across them, working with original teams to get the nuances right. So they had larger samples than originals and ran experiments multiple times across multiple samples.
Where a study failed in one it mostly failed everywhere, and where it worked it mostly worked in all the experiments.
So, yes, priming someone with the number 32 won't get them to bet it in the casino, but it does mean that social science can be experimental - you can build and run lab scale social science - a positive !
- ALittleLight 8y agoThis is a really optimistic take. I agree with you, ultimately, this is positive because it shows that it's possible for psychology to be done and have widespread meaningful results. In the short term though this seems extremely negative as it indicates that the psychology field and researchers we have today are producing substandard results. Two parts of this article increase my negative feelings. First, they mention that these 28 studies were selected because they were well known and influential. If well known and influential studies only have a 50% reproducibility rate - what about the average study? Another point I find very troubling is that people betting online were able to predict, with high accuracy, which studies were accurate and which weren't. This tells me that the problem is an "Emperor's New Clothes" type of problem and not just a challenge with the fundamental difficulty of the subject since the subject is so yielding to the public's intuition. Instead of a problem with the fundamental difficulty, the problem seems to be with the psychology of the researchers - being unable to diagnose and prevent methodological flaws in their research. This class of failure is much less excusable to me.
- lifeisstillgood 8y agoFor me the unrepeatable experiments suggest something even more interesting - that unless the originals were flawed or fraud, then something about the people tested has changed. We are all more aware of conmen movies, of fake news, or have read popsci books and now know what to look out for.
- lifeisstillgood 8y agook having looked deeper into this it seems some of the i replicated studies are ... counter intuitive.
- jonex 8y ago"If well known and influential studies only have a 50% reproducibility rate - what about the average study?" Actually, they might be better off. In the related article* they mention: "Beyond statistical issues, it strikes me that several of the studies that didn’t replicate have another quality in common: newsworthiness." It could be that many well known works are well known because their finding are counterintuitive, and therefore more interesting. However, this could also imply that the prior probability is lower - hence a p = 0.05 result is more likely to be due to chance than for a study finding something more in line with prior understanding of things. * https://www.theatlantic.com/science/archive/2018/08/scientists-can-collectively-sense-which-psychology-studies-are-weak/568630/ https://www.theatlantic.com/science/archive/2018/08/scientis...
- repolfx 8y agoThe article that goes deeper into the betting markets says that they suspect many of the traders were people invested in the reproducibility crisis and related research, that is, they are more knowledgable than the general public would be. That doesn't take away from your point, but the problem here may be more refined than "the public" vs "psychologists". It may be more like "some psychologists" vs "journal editors and other psychologists".
- ALittleLight 8y agoWhat I wrote was influenced by the fact that I had taken a "Psychology Replication Quiz" which I'll link to below. The quiz presents you with the thesis of a study and asks whether you think it will replicate. When I took it the first time I got ten questions correct out of ten. I took the quiz again a moment before writing this comment and got nine out of ten (I think I was hurrying this time). The average person taking the quiz gets seven out ten correct. The depressing part about this quiz is how easy it is. You just have to think for a moment if what they are describing matches what you know about human nature. Some statements just feel very intuitive and likely, and those replicate. Other statements seem quite wild and abstract, and those don't replicate. I haven't seen a list of the 28 papers they looked at here, but I'd bet it is similar. Some of the findings are incredible (and not reproducible) while others are intuitive and reproducible. I'd encourage you to take the quiz below and see if you feel the same way about the results as I do. Psychology Replication Quiz - https://80000hours.org/psychology-replication-quiz/ https://80000hours.org/psychology-replication-quiz/
- djsumdog 8y agoI wish there was more of this optimism. Yes, it sends a lot of conceptual ideas back to the drawing board, but that just means there are all types of new experiments to be composed and preformed. A number of these experiments did replicate, and have such a huge set in multiple regions/culture is really really important. Say this: in 120 years some of those experiments in the positive replicated set failed. It won't be as easy to just contribute that to sloppy findings. It could me the culture of human beings has changed, as well as our underlying psychological tendencies. If it was just two or three studies, you could dismiss them and say they might have been wrong, but with this data set, you could say with higher confidence that it's more likely something about our society has changed.
- redler 8y agoI also read it as a qualified positive. (1) Sound and unsound studies can be done in any field. (2) In this examination of the field of psychology, half the observed studies were found to be sound. (3) The implication that half the work in psychology may be unsound is concerning, and suggests a large cohort with problems in rigor, ethics, careerism, etc. (4) The sound work being done in psychology is as sound as the work in other sciences, and it's the unsound work that's the problem. (5) As an interesting side note, unsoundness can be intuited to a statistically significant degree.