3 ms·

On the psychology replication crisis >Overall, 36% of the replications yielded significant findings (p value below 0.05) compared to 97% of the original studie

On the psychology replication crisis >Overall, 36% of the replications yielded significant findings (p value below 0.05) compared to 97% of the original studies that had significant effects. The mean effect size in the replications was approximately half the magnitude of the effects reported in the original studies. https://en.wikipedia.org/wiki/Replication_crisis#Psychology_replication_rates https://en.wikipedia.org/wiki/Replication_crisis#Psychology_... Of course if you point out that psychology research is strongly biased, the immediate response is that you must be an anti-intellectual. As if the system that produced such misleading results is a representation of all intellectual pursuits, and not just a mistake. The same way you get called an anti-intellectual for questioning the post-modern-analyses of whatever. I'm really sick of people claiming that criticizing obviously bad science is "anti-intellectual". When you produce that many useless papers you're obviously, as a field, lacking an understanding of why science is good and useful in the first place. I'm presuming that economics has similar problems with replication, and that you can only trust the most basic and obvious of their findings.

sarbaz 8y ago
A lot of this is simply that p < 0.05 is a very low bar for research. If we want to understand how the world works, we need to be able to construct mechanisms that explain why the results we measure in experiments come about. Without this thing, it's not science. A good mechanism that accurately explains some part of the world can be validated with a much better result than p < 0.05, because it can be properly isolated from other factors and its effect size can be strengthened. Without a mechanism to explain how things work, the experimental results that get published are probably just noise. And because p < 0.05 is such a low bar to pass, lots of noise gets published. Trying to reproduce an experiment whose results were drawn from /dev/urandom is pointless. In fact I'm surprised the replication was so high. I was expecting it to be around 5% (i.e. results are totally random). So maybe there is some hidden merit to all of this
- nicoburns 8y ago
  > If we want to understand how the world works, we need to be able to construct mechanisms that explain why the results we measure in experiments come about. This, 1000 times over. Unfortunately we seem to teach that statistical analysis is sufficient to infer causation in a lot our undergraduate science programs.
- traverseda 8y ago
  I wouldn't expect totally random because people have common sense. I'd expect it to be similar to the folk-wisdom of the time, since a lot of it is going to be repackaging of that same folk-wisdom, and negative results tend not to get published, or get massaged until they're positive. I'd expect social-psychology to mirror the dominant opinions of people in other segments of the humanities, although it is of course hard to determine the direction of that relationship.
deleted 8y ago
[deleted]
UncleMeat 8y ago
A serious problem is when people assume that because the replication crisis in psych is widely publicized that it is exclusive to psych and then they dismiss all of the findings in the field. In my experience the replication crisis is rarely used to criticize method but instead used to dismiss research out of hand, whether or not it is supported by other research that limits the likelihood of error. I think that psych should be commended for attacking this problem. As a field they are funding replications, which basically never happens in any other field. We don't even know what the "replication crisis" would be in empirical CS. Given discussions with ML people I'd wager that the "replication crisis" in ML is at least as dramatic. Science is messy. But this is why we should defer to experts who have vision over an entire field rather than focusing on individual papers as conclusive data points.
iron0012 8y ago
I'm not sure what the replication crisis (which as others have noted is certainly not exclusive to psychology--they're just the only field that is acknowledging it and taking steps to address it) has to do with bias (by which you seem to mean political bias, not statistical bias which is very different). Your attack on what you perceive to be bias on the part of researchers really seems like a non sequitur with respect to replication.
spruciefic399 8y ago
To be consistent, you'd have to leverage your arguments against other fields with replication crises, which include neuroscience and oncology. It's really true of the biomedical sciences in general. Psychology has a history of being sort of self-analytical (which makes sense if you're a field studying human behavior). Meta-analysis, for example, was largely incubated in psychology before it started expanding into other disciplines. The criticism I'm aware of is really being leveled at the idea that this is somehow unique to psychology. It's like killing the messenger, or someone who is trying to fix the problem.

sarbaz 8y ago

A lot of this is simply that p < 0.05 is a very low bar for research. If we want to understand how the world works, we need to be able to construct mechanisms that explain why the results we measure in experiments come about. Without this thing, it's not science. A good mechanism that accurately explains some part of the world can be validated with a much better result than p < 0.05, because it can be properly isolated from other factors and its effect size can be strengthened. Without a mechanism to explain how things work, the experimental results that get published are probably just noise. And because p < 0.05 is such a low bar to pass, lots of noise gets published. Trying to reproduce an experiment whose results were drawn from /dev/urandom is pointless. In fact I'm surprised the replication was so high. I was expecting it to be around 5% (i.e. results are totally random). So maybe there is some hidden merit to all of this

nicoburns 8y ago
> If we want to understand how the world works, we need to be able to construct mechanisms that explain why the results we measure in experiments come about. This, 1000 times over. Unfortunately we seem to teach that statistical analysis is sufficient to infer causation in a lot our undergraduate science programs.
traverseda 8y ago
I wouldn't expect totally random because people have common sense. I'd expect it to be similar to the folk-wisdom of the time, since a lot of it is going to be repackaging of that same folk-wisdom, and negative results tend not to get published, or get massaged until they're positive. I'd expect social-psychology to mirror the dominant opinions of people in other segments of the humanities, although it is of course hard to determine the direction of that relationship.

deleted 8y ago

[deleted]

UncleMeat 8y ago

A serious problem is when people assume that because the replication crisis in psych is widely publicized that it is exclusive to psych and then they dismiss all of the findings in the field. In my experience the replication crisis is rarely used to criticize method but instead used to dismiss research out of hand, whether or not it is supported by other research that limits the likelihood of error. I think that psych should be commended for attacking this problem. As a field they are funding replications, which basically never happens in any other field. We don't even know what the "replication crisis" would be in empirical CS. Given discussions with ML people I'd wager that the "replication crisis" in ML is at least as dramatic. Science is messy. But this is why we should defer to experts who have vision over an entire field rather than focusing on individual papers as conclusive data points.

iron0012 8y ago

I'm not sure what the replication crisis (which as others have noted is certainly not exclusive to psychology--they're just the only field that is acknowledging it and taking steps to address it) has to do with bias (by which you seem to mean political bias, not statistical bias which is very different). Your attack on what you perceive to be bias on the part of researchers really seems like a non sequitur with respect to replication.

spruciefic399 8y ago

To be consistent, you'd have to leverage your arguments against other fields with replication crises, which include neuroscience and oncology. It's really true of the biomedical sciences in general. Psychology has a history of being sort of self-analytical (which makes sense if you're a field studying human behavior). Meta-analysis, for example, was largely incubated in psychology before it started expanding into other disciplines. The criticism I'm aware of is really being leveled at the idea that this is somehow unique to psychology. It's like killing the messenger, or someone who is trying to fix the problem.