4 ms·

I think that's an exaggeration. The replication crisis wasn't just this one paper. There were also (e.g.) Daniel Bem's studies on ESP and a whole bunch of indiv

by dash2 4y ago

I think that's an exaggeration. The replication crisis wasn't just this one paper. There were also (e.g.) Daniel Bem's studies on ESP and a whole bunch of individual failed replications. The point of the post is that the size of the replication crisis may have been exaggerated in this study.

wolverine876 4y ago
> The replication crisis wasn't just this one paper. There were also (e.g.) Daniel Bem's studies on ESP and a whole bunch of individual failed replications. That's not a crisis, if I understand correctly what you mean. Failed replication is an essential part of science, like bankruptcy is an essential part of business. The reason the OSC replication failure paper was important was that it discussed widely accepted research. > The point of the post is that the size of the replication crisis may have been exaggerated in this study. The OP is much stronger than that. For example, First, they introduced statistical error into the data which led the OSC to significantly underestimate how many of their replications should have failed by chance alone. When this error is taken into account, the number of failures in their data is no greater than one would expect if all 100 of the original findings had been true. and “What they did,” added Gilbert, “is create an idiosyncratic, arbitrary list of sampling rules that excluded the majority of psychology’s subfields from the sample, that excluded entire classes of studies whose methods are probably among the best in science from the sample, and so on. Then they proceeded to violate all of their own rules. ...” and “The replicators realized that doing this study [using people speaking English about US social issues] in the Netherlands might have been a problem, so they wisely decided to run another version of it in the US. And when they did, they basically replicated the original result. And yet, when the OSC estimated the reproducibility of psychological science, they excluded the successful replication and included only the one from the University of Amsterdam that failed. ...”
- uniqueuid 4y ago
  I think we still agree that replications are almost never done in psychology and social science, and when done they are almost never published. This is compared to lab-based fields such as biology. Also, mandatory pre-registrations provide some evidence that effects are exaggerated in other fields as well, see e.g. [1] for treatments for cardiovascular diseases. There is a crisis on many fronts, but if you prefer we can call it "professionalization, finally". [1] https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0132382 https://journals.plos.org/plosone/article?id=10.1371/journal...
  - wolverine876 4y ago
    > I think we still agree that replications are almost never done in psychology and social science, and when done they are almost never published. This is compared to lab-based fields such as biology. If that's true, could it be a matter of funding? Social sciences receive much less, afaik. I remember going into an experimental psychology building in a major university that you've certainly heard of, though it was years ago. It was an old K-12 school, scarsely remodeled, and files were stuffed in lockers.
- dash2 4y ago
  > That's not a crisis, if I understand correctly what you mean. Failed replication is an essential part of science.... Right, but the replication crisis is a broad term for a bunch of people, in a bunch of disciplines, worrying that their work might not replicate for reasons like p hacking, forking paths, etc. I mentioned those particular studies because they made people sit up and go "hey, there's a bigger problem here". The OSC paper came out of this situation, it didn't start it. Other symptoms include that spoof paper where listening to newer music "made participants younger". Andrew Gelman's blog. Etc. Etc.
  - wolverine876 4y ago
    > the replication crisis is a broad term for a bunch of people, in a bunch of disciplines, worrying that their work might not replicate for reasons like p hacking, forking paths, etc. Is there something serious - non-partisan, sophisticated, informed - that covers this? Gelman's blog mostly argues by dismissive tone, sadly (though people can't seem to get enough of that on the Internet!).
    - dash2 4y ago
      Gelman's tone may be dismissive, but he's a serious statistician and is taken seriously by the community, I think.
      - wolverine876 4y ago
        > he's a serious statistician and is taken seriously by the community Absolutely and I'm sure Gelman's scholarly work is taken seriously. But as Gelman says: Brilliance represents an upper bound on the quality of your reasoning, but there is no lower bound. The most brilliant scientist in the world can take really dumb stances. ... The best driver in the world can still decide to turn the steering wheel and crash into a tree. [0] I don't think the blog is at the upper bound. I think many - including many smart, very human people - get taken in by path of (unfortunately) socially normative Internet rhetoric - it's easy to follow (requires conscious effort to avoid) and socially rewarding - which is heavy on contempt and outrage as the primary forces, as if they indicated truth or value, and not evidence and reason. [0] https://statmodeling.stat.columbia.edu/2022/06/08/how-can-a-top-scientist-be-so-confidently-wrong-r-a-fisher-and-smoking-example/ https://statmodeling.stat.columbia.edu/2022/06/08/how-can-a-...

by dash2 4y ago

wolverine876 4y ago

> The replication crisis wasn't just this one paper. There were also (e.g.) Daniel Bem's studies on ESP and a whole bunch of individual failed replications. That's not a crisis, if I understand correctly what you mean. Failed replication is an essential part of science, like bankruptcy is an essential part of business. The reason the OSC replication failure paper was important was that it discussed widely accepted research. > The point of the post is that the size of the replication crisis may have been exaggerated in this study. The OP is much stronger than that. For example, First, they introduced statistical error into the data which led the OSC to significantly underestimate how many of their replications should have failed by chance alone. When this error is taken into account, the number of failures in their data is no greater than one would expect if all 100 of the original findings had been true. and “What they did,” added Gilbert, “is create an idiosyncratic, arbitrary list of sampling rules that excluded the majority of psychology’s subfields from the sample, that excluded entire classes of studies whose methods are probably among the best in science from the sample, and so on. Then they proceeded to violate all of their own rules. ...” and “The replicators realized that doing this study [using people speaking English about US social issues] in the Netherlands might have been a problem, so they wisely decided to run another version of it in the US. And when they did, they basically replicated the original result. And yet, when the OSC estimated the reproducibility of psychological science, they excluded the successful replication and included only the one from the University of Amsterdam that failed. ...”

uniqueuid 4y ago
I think we still agree that replications are almost never done in psychology and social science, and when done they are almost never published. This is compared to lab-based fields such as biology. Also, mandatory pre-registrations provide some evidence that effects are exaggerated in other fields as well, see e.g. [1] for treatments for cardiovascular diseases. There is a crisis on many fronts, but if you prefer we can call it "professionalization, finally". [1] https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0132382 https://journals.plos.org/plosone/article?id=10.1371/journal...
- wolverine876 4y ago
  > I think we still agree that replications are almost never done in psychology and social science, and when done they are almost never published. This is compared to lab-based fields such as biology. If that's true, could it be a matter of funding? Social sciences receive much less, afaik. I remember going into an experimental psychology building in a major university that you've certainly heard of, though it was years ago. It was an old K-12 school, scarsely remodeled, and files were stuffed in lockers.
dash2 4y ago
> That's not a crisis, if I understand correctly what you mean. Failed replication is an essential part of science.... Right, but the replication crisis is a broad term for a bunch of people, in a bunch of disciplines, worrying that their work might not replicate for reasons like p hacking, forking paths, etc. I mentioned those particular studies because they made people sit up and go "hey, there's a bigger problem here". The OSC paper came out of this situation, it didn't start it. Other symptoms include that spoof paper where listening to newer music "made participants younger". Andrew Gelman's blog. Etc. Etc.
- wolverine876 4y ago
  > the replication crisis is a broad term for a bunch of people, in a bunch of disciplines, worrying that their work might not replicate for reasons like p hacking, forking paths, etc. Is there something serious - non-partisan, sophisticated, informed - that covers this? Gelman's blog mostly argues by dismissive tone, sadly (though people can't seem to get enough of that on the Internet!).
  - dash2 4y ago
    Gelman's tone may be dismissive, but he's a serious statistician and is taken seriously by the community, I think.
    - wolverine876 4y ago
      > he's a serious statistician and is taken seriously by the community Absolutely and I'm sure Gelman's scholarly work is taken seriously. But as Gelman says: Brilliance represents an upper bound on the quality of your reasoning, but there is no lower bound. The most brilliant scientist in the world can take really dumb stances. ... The best driver in the world can still decide to turn the steering wheel and crash into a tree. [0] I don't think the blog is at the upper bound. I think many - including many smart, very human people - get taken in by path of (unfortunately) socially normative Internet rhetoric - it's easy to follow (requires conscious effort to avoid) and socially rewarding - which is heavy on contempt and outrage as the primary forces, as if they indicated truth or value, and not evidence and reason. [0] https://statmodeling.stat.columbia.edu/2022/06/08/how-can-a-top-scientist-be-so-confidently-wrong-r-a-fisher-and-smoking-example/ https://statmodeling.stat.columbia.edu/2022/06/08/how-can-a-...