4 ms·

> But many disciplines, in this case psychology, did have the replication crisis coming. The point of the OP is that there isn't a replication crisis. Should w

> But many disciplines, in this case psychology, did have the replication crisis coming. The point of the OP is that there isn't a replication crisis. Should we be assuming, for purposes of this discussion, that there is one?

dash2 4y ago
I think that's an exaggeration. The replication crisis wasn't just this one paper. There were also (e.g.) Daniel Bem's studies on ESP and a whole bunch of individual failed replications. The point of the post is that the size of the replication crisis may have been exaggerated in this study.
- wolverine876 4y ago
  > The replication crisis wasn't just this one paper. There were also (e.g.) Daniel Bem's studies on ESP and a whole bunch of individual failed replications. That's not a crisis, if I understand correctly what you mean. Failed replication is an essential part of science, like bankruptcy is an essential part of business. The reason the OSC replication failure paper was important was that it discussed widely accepted research. > The point of the post is that the size of the replication crisis may have been exaggerated in this study. The OP is much stronger than that. For example, First, they introduced statistical error into the data which led the OSC to significantly underestimate how many of their replications should have failed by chance alone. When this error is taken into account, the number of failures in their data is no greater than one would expect if all 100 of the original findings had been true. and “What they did,” added Gilbert, “is create an idiosyncratic, arbitrary list of sampling rules that excluded the majority of psychology’s subfields from the sample, that excluded entire classes of studies whose methods are probably among the best in science from the sample, and so on. Then they proceeded to violate all of their own rules. ...” and “The replicators realized that doing this study [using people speaking English about US social issues] in the Netherlands might have been a problem, so they wisely decided to run another version of it in the US. And when they did, they basically replicated the original result. And yet, when the OSC estimated the reproducibility of psychological science, they excluded the successful replication and included only the one from the University of Amsterdam that failed. ...”
  - uniqueuid 4y ago
    I think we still agree that replications are almost never done in psychology and social science, and when done they are almost never published. This is compared to lab-based fields such as biology. Also, mandatory pre-registrations provide some evidence that effects are exaggerated in other fields as well, see e.g. [1] for treatments for cardiovascular diseases. There is a crisis on many fronts, but if you prefer we can call it "professionalization, finally". [1] https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0132382 https://journals.plos.org/plosone/article?id=10.1371/journal...
    - wolverine876 4y ago
      > I think we still agree that replications are almost never done in psychology and social science, and when done they are almost never published. This is compared to lab-based fields such as biology. If that's true, could it be a matter of funding? Social sciences receive much less, afaik. I remember going into an experimental psychology building in a major university that you've certainly heard of, though it was years ago. It was an old K-12 school, scarsely remodeled, and files were stuffed in lockers.
  - dash2 4y ago
    > That's not a crisis, if I understand correctly what you mean. Failed replication is an essential part of science.... Right, but the replication crisis is a broad term for a bunch of people, in a bunch of disciplines, worrying that their work might not replicate for reasons like p hacking, forking paths, etc. I mentioned those particular studies because they made people sit up and go "hey, there's a bigger problem here". The OSC paper came out of this situation, it didn't start it. Other symptoms include that spoof paper where listening to newer music "made participants younger". Andrew Gelman's blog. Etc. Etc.
    - wolverine876 4y ago
      > the replication crisis is a broad term for a bunch of people, in a bunch of disciplines, worrying that their work might not replicate for reasons like p hacking, forking paths, etc. Is there something serious - non-partisan, sophisticated, informed - that covers this? Gelman's blog mostly argues by dismissive tone, sadly (though people can't seem to get enough of that on the Internet!).
kryptiskt 4y ago
This paper drew heavy criticism at the time: http://retractionwatch.com/2016/03/07/lets-not-mischaracterize-replication-studies-authors/ http://retractionwatch.com/2016/03/07/lets-not-mischaracteri... And the years since then haven't been kind to landmark results in psychology.
- wolverine876 4y ago
  > the years since then haven't been kind to landmark results in psychology I've seen those criticisms repeated many times, but IME each time it lacks evidence. We know that is arguably the primary form of misinformation (and disinformation) - repeating false or unfounded assertions until people assume they are true - so the fact that it is accepted by some or repeated or assumed doesn't make it true. (In fact, science was created to address that exact problem.) EDIT: > This paper drew heavy criticism at the time: http://retractionwatch.com http://retractionwatch.com ... The link is to a response by the authors of the original paper. They certainly should be heard, but they are not representative of heavy criticism in the field.

dash2 4y ago

I think that's an exaggeration. The replication crisis wasn't just this one paper. There were also (e.g.) Daniel Bem's studies on ESP and a whole bunch of individual failed replications. The point of the post is that the size of the replication crisis may have been exaggerated in this study.

wolverine876 4y ago
> The replication crisis wasn't just this one paper. There were also (e.g.) Daniel Bem's studies on ESP and a whole bunch of individual failed replications. That's not a crisis, if I understand correctly what you mean. Failed replication is an essential part of science, like bankruptcy is an essential part of business. The reason the OSC replication failure paper was important was that it discussed widely accepted research. > The point of the post is that the size of the replication crisis may have been exaggerated in this study. The OP is much stronger than that. For example, First, they introduced statistical error into the data which led the OSC to significantly underestimate how many of their replications should have failed by chance alone. When this error is taken into account, the number of failures in their data is no greater than one would expect if all 100 of the original findings had been true. and “What they did,” added Gilbert, “is create an idiosyncratic, arbitrary list of sampling rules that excluded the majority of psychology’s subfields from the sample, that excluded entire classes of studies whose methods are probably among the best in science from the sample, and so on. Then they proceeded to violate all of their own rules. ...” and “The replicators realized that doing this study [using people speaking English about US social issues] in the Netherlands might have been a problem, so they wisely decided to run another version of it in the US. And when they did, they basically replicated the original result. And yet, when the OSC estimated the reproducibility of psychological science, they excluded the successful replication and included only the one from the University of Amsterdam that failed. ...”
- uniqueuid 4y ago
  I think we still agree that replications are almost never done in psychology and social science, and when done they are almost never published. This is compared to lab-based fields such as biology. Also, mandatory pre-registrations provide some evidence that effects are exaggerated in other fields as well, see e.g. [1] for treatments for cardiovascular diseases. There is a crisis on many fronts, but if you prefer we can call it "professionalization, finally". [1] https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0132382 https://journals.plos.org/plosone/article?id=10.1371/journal...
  - wolverine876 4y ago
    > I think we still agree that replications are almost never done in psychology and social science, and when done they are almost never published. This is compared to lab-based fields such as biology. If that's true, could it be a matter of funding? Social sciences receive much less, afaik. I remember going into an experimental psychology building in a major university that you've certainly heard of, though it was years ago. It was an old K-12 school, scarsely remodeled, and files were stuffed in lockers.
- dash2 4y ago
  > That's not a crisis, if I understand correctly what you mean. Failed replication is an essential part of science.... Right, but the replication crisis is a broad term for a bunch of people, in a bunch of disciplines, worrying that their work might not replicate for reasons like p hacking, forking paths, etc. I mentioned those particular studies because they made people sit up and go "hey, there's a bigger problem here". The OSC paper came out of this situation, it didn't start it. Other symptoms include that spoof paper where listening to newer music "made participants younger". Andrew Gelman's blog. Etc. Etc.
  - wolverine876 4y ago
    > the replication crisis is a broad term for a bunch of people, in a bunch of disciplines, worrying that their work might not replicate for reasons like p hacking, forking paths, etc. Is there something serious - non-partisan, sophisticated, informed - that covers this? Gelman's blog mostly argues by dismissive tone, sadly (though people can't seem to get enough of that on the Internet!).

kryptiskt 4y ago

This paper drew heavy criticism at the time: http://retractionwatch.com/2016/03/07/lets-not-mischaracterize-replication-studies-authors/ http://retractionwatch.com/2016/03/07/lets-not-mischaracteri... And the years since then haven't been kind to landmark results in psychology.

wolverine876 4y ago
> the years since then haven't been kind to landmark results in psychology I've seen those criticisms repeated many times, but IME each time it lacks evidence. We know that is arguably the primary form of misinformation (and disinformation) - repeating false or unfounded assertions until people assume they are true - so the fact that it is accepted by some or repeated or assumed doesn't make it true. (In fact, science was created to address that exact problem.) EDIT: > This paper drew heavy criticism at the time: http://retractionwatch.com http://retractionwatch.com ... The link is to a response by the authors of the original paper. They certainly should be heard, but they are not representative of heavy criticism in the field.