4 ms·

First, this is [2016]. Second, as ethanbond has commented, a key question in replication is how strictly one should interpret study designs and how much agreem

First, this is [2016]. Second, as ethanbond has commented, a key question in replication is how strictly one should interpret study designs and how much agreement one would expect. I do believe though that there is a true replication crisis, not just because the situational context is different (i.e. generalization) but also because a lot of the original studies were unintentionally or intentionally wrong, had way too little power, i.e. noisy measurements, small N and publication bias. Recent advances in the field of replicability include specification curve analyses, for example, and those have been able to show more conclusive evidence of non-replicability because the published results were unbelievably special (i.e. one out of a tiny fraction of significant results).

wolverine876 4y ago
> a lot of the original studies were unintentionally or intentionally wrong, had way too little power, i.e. noisy measurements, small N and publication bias. That seems much too obvious for all these scientists to overlook, including the OP. Do the authors of the original replication paper say that? > Recent advances in the field of replicability include specification curve analyses, for example, and those have been able to show more conclusive evidence of non-replicability because the published results were unbelievably special (i.e. one out of a tiny fraction of significant results). Would you have links to this research? Thanks.
- uniqueuid 4y ago
  https://www.nature.com/articles/s41562-020-0912-z https://www.nature.com/articles/s41562-020-0912-z Simonsohn and Simmons have a couple of great other papers as well. Also: Sure, many of the problems in sciences are obvious, just as many of the problems in other parts of society are. That doesn't mean that it's rational for most people to care about them. Plus there are some hard problems very adjacent to easy problems, and that's an invitation to throw them all in a bucket and not care. Examples: Low N is easy to recognize, but near impossible to change when studying rare diseases. Non-linear phenomena in linear statistical tests are easy to spot, but incompleteness of models (key valuables omitted) or specification errors in structural equation models are hard to spot (need lots of domain expertise). [edit]: If you want to do a spec curve analysis, try this easy package https://cran.r-project.org/web//packages/specr/index.html https://cran.r-project.org/web//packages/specr/index.html

wolverine876 4y ago

> a lot of the original studies were unintentionally or intentionally wrong, had way too little power, i.e. noisy measurements, small N and publication bias. That seems much too obvious for all these scientists to overlook, including the OP. Do the authors of the original replication paper say that? > Recent advances in the field of replicability include specification curve analyses, for example, and those have been able to show more conclusive evidence of non-replicability because the published results were unbelievably special (i.e. one out of a tiny fraction of significant results). Would you have links to this research? Thanks.

uniqueuid 4y ago
https://www.nature.com/articles/s41562-020-0912-z https://www.nature.com/articles/s41562-020-0912-z Simonsohn and Simmons have a couple of great other papers as well. Also: Sure, many of the problems in sciences are obvious, just as many of the problems in other parts of society are. That doesn't mean that it's rational for most people to care about them. Plus there are some hard problems very adjacent to easy problems, and that's an invitation to throw them all in a bucket and not care. Examples: Low N is easy to recognize, but near impossible to change when studying rare diseases. Non-linear phenomena in linear statistical tests are easy to spot, but incompleteness of models (key valuables omitted) or specification errors in structural equation models are hard to spot (need lots of domain expertise). [edit]: If you want to do a spec curve analysis, try this easy package https://cran.r-project.org/web//packages/specr/index.html https://cran.r-project.org/web//packages/specr/index.html