5 ms·

I agree that a nuanced approach is the right way. But many disciplines, in this case psychology, did have the replication crisis coming. Cohen and Gigerenzer h

I agree that a nuanced approach is the right way. But many disciplines, in this case psychology, did have the replication crisis coming. Cohen and Gigerenzer have warned about power for literally decades but their warnings were not heeded by a significant part of the field. So a stark message may in fact have been necessary. By the way, Daniel Kahneman basically wrote a "this is a problem, deal with it NOW" letter to some colleagues in 2012 [1]. Very telling and an insightful read. [1] http://www.decisionsciencenews.com/?p=3634 http://www.decisionsciencenews.com/?p=3634

jvanderbot 4y ago
Here's what he suggested: * Assemble a group of five labs, where the leading investigators have an established reputation (tenure should perhaps be a requirement). Substantial labs with several students are the most desirable participants. * Each lab selects a recent demonstration of a priming effect, which they consider robust and most likely to replicate. * The board makes a public commitment to these five specific effects * Set up a daisy chain of labs A-B-C-D-E-A, where each lab will replicate the study selected by its neighbor: B replicates A, C replicates B etc. * Have the replicating lab send someone to see how subjects are run (hence the emphasis on recency – the experiments should be in the active repertoire of the original lab, so that additional subjects can be run with confidence that the same procedure is followed). * Have the replicated lab send someone to vet the procedure of the replicating lab as it starts its work * Run enough subjects to guarantee power (probably more than in the original study) * Use technology (e.g. video) to ensure that every detail of the method is documented and can be copied by others. * Pre-commit to publish the results, letting the chips fall where they may, and make all data available for analysis by others. I hope that psychology is a field where it is possible to fund such a protocol from grants or from internal school funds. However, that protocol seems to me to be laborious, expensive, and not at all "easy to get".
wolverine876 4y ago
> But many disciplines, in this case psychology, did have the replication crisis coming. The point of the OP is that there isn't a replication crisis. Should we be assuming, for purposes of this discussion, that there is one?
- dash2 4y ago
  I think that's an exaggeration. The replication crisis wasn't just this one paper. There were also (e.g.) Daniel Bem's studies on ESP and a whole bunch of individual failed replications. The point of the post is that the size of the replication crisis may have been exaggerated in this study.
  - wolverine876 4y ago
    > The replication crisis wasn't just this one paper. There were also (e.g.) Daniel Bem's studies on ESP and a whole bunch of individual failed replications. That's not a crisis, if I understand correctly what you mean. Failed replication is an essential part of science, like bankruptcy is an essential part of business. The reason the OSC replication failure paper was important was that it discussed widely accepted research. > The point of the post is that the size of the replication crisis may have been exaggerated in this study. The OP is much stronger than that. For example, First, they introduced statistical error into the data which led the OSC to significantly underestimate how many of their replications should have failed by chance alone. When this error is taken into account, the number of failures in their data is no greater than one would expect if all 100 of the original findings had been true. and “What they did,” added Gilbert, “is create an idiosyncratic, arbitrary list of sampling rules that excluded the majority of psychology’s subfields from the sample, that excluded entire classes of studies whose methods are probably among the best in science from the sample, and so on. Then they proceeded to violate all of their own rules. ...” and “The replicators realized that doing this study [using people speaking English about US social issues] in the Netherlands might have been a problem, so they wisely decided to run another version of it in the US. And when they did, they basically replicated the original result. And yet, when the OSC estimated the reproducibility of psychological science, they excluded the successful replication and included only the one from the University of Amsterdam that failed. ...”
    - uniqueuid 4y ago
      I think we still agree that replications are almost never done in psychology and social science, and when done they are almost never published. This is compared to lab-based fields such as biology. Also, mandatory pre-registrations provide some evidence that effects are exaggerated in other fields as well, see e.g. [1] for treatments for cardiovascular diseases. There is a crisis on many fronts, but if you prefer we can call it "professionalization, finally". [1] https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0132382 https://journals.plos.org/plosone/article?id=10.1371/journal...

jvanderbot 4y ago

Here's what he suggested: * Assemble a group of five labs, where the leading investigators have an established reputation (tenure should perhaps be a requirement). Substantial labs with several students are the most desirable participants. * Each lab selects a recent demonstration of a priming effect, which they consider robust and most likely to replicate. * The board makes a public commitment to these five specific effects * Set up a daisy chain of labs A-B-C-D-E-A, where each lab will replicate the study selected by its neighbor: B replicates A, C replicates B etc. * Have the replicating lab send someone to see how subjects are run (hence the emphasis on recency – the experiments should be in the active repertoire of the original lab, so that additional subjects can be run with confidence that the same procedure is followed). * Have the replicated lab send someone to vet the procedure of the replicating lab as it starts its work * Run enough subjects to guarantee power (probably more than in the original study) * Use technology (e.g. video) to ensure that every detail of the method is documented and can be copied by others. * Pre-commit to publish the results, letting the chips fall where they may, and make all data available for analysis by others. I hope that psychology is a field where it is possible to fund such a protocol from grants or from internal school funds. However, that protocol seems to me to be laborious, expensive, and not at all "easy to get".

wolverine876 4y ago

> But many disciplines, in this case psychology, did have the replication crisis coming. The point of the OP is that there isn't a replication crisis. Should we be assuming, for purposes of this discussion, that there is one?

dash2 4y ago
I think that's an exaggeration. The replication crisis wasn't just this one paper. There were also (e.g.) Daniel Bem's studies on ESP and a whole bunch of individual failed replications. The point of the post is that the size of the replication crisis may have been exaggerated in this study.
- wolverine876 4y ago
  > The replication crisis wasn't just this one paper. There were also (e.g.) Daniel Bem's studies on ESP and a whole bunch of individual failed replications. That's not a crisis, if I understand correctly what you mean. Failed replication is an essential part of science, like bankruptcy is an essential part of business. The reason the OSC replication failure paper was important was that it discussed widely accepted research. > The point of the post is that the size of the replication crisis may have been exaggerated in this study. The OP is much stronger than that. For example, First, they introduced statistical error into the data which led the OSC to significantly underestimate how many of their replications should have failed by chance alone. When this error is taken into account, the number of failures in their data is no greater than one would expect if all 100 of the original findings had been true. and “What they did,” added Gilbert, “is create an idiosyncratic, arbitrary list of sampling rules that excluded the majority of psychology’s subfields from the sample, that excluded entire classes of studies whose methods are probably among the best in science from the sample, and so on. Then they proceeded to violate all of their own rules. ...” and “The replicators realized that doing this study [using people speaking English about US social issues] in the Netherlands might have been a problem, so they wisely decided to run another version of it in the US. And when they did, they basically replicated the original result. And yet, when the OSC estimated the reproducibility of psychological science, they excluded the successful replication and included only the one from the University of Amsterdam that failed. ...”
  - uniqueuid 4y ago
    I think we still agree that replications are almost never done in psychology and social science, and when done they are almost never published. This is compared to lab-based fields such as biology. Also, mandatory pre-registrations provide some evidence that effects are exaggerated in other fields as well, see e.g. [1] for treatments for cardiovascular diseases. There is a crisis on many fronts, but if you prefer we can call it "professionalization, finally". [1] https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0132382 https://journals.plos.org/plosone/article?id=10.1371/journal...