15 ms·

With some of the things, but admittedly not most of the things you mentioned, there's a dataset (somewhere) and some code run on that dataset (somewhere) and re

by msla 3y ago

With some of the things, but admittedly not most of the things you mentioned, there's a dataset (somewhere) and some code run on that dataset (somewhere) and replication would mean someone else being able to run that code on that dataset and get the same results. Would this require labs to improve their software environments and learn some new tools? Would this require labs to give up whatever used to be secret sauce? That's. The. Point.

counters 3y ago
In practice this is happening in many disciplines, for most research, on a daily basis. What _isn't_ happening is that the results of these replications are being independently peer reviewed, because that isn't incentivized. However, when replication fails for whatever reason, it usually leads to insights that themselves lead to stronger scientific work and better publications later on.
eesmith 3y ago
> someone else being able to run that code on that dataset and get the same results. I think when people talk about "replicate" they mean something more than that. The dataset could contain coding errors, and the analysis could contain incorrect formulas and bad modeling. Reproducing a bad analysis, successfully, provide no corrective feedback. I know for one paper I could replicate the paper's results using the paper's own analysis, but I couldn't replicate the paper's results using my analysis. > Would this require labs to give up whatever used to be secret sauce? That's. The. Point. That seems to be a very different Point. Newton famously published results made from using his secret sauce - calculus - by recasting them using more traditional methods. In the extreme cas, I could publish the factors for RSA-1024 without publishing my factorization method. "I prayed to God for the answer and He gave them to me." You can verify that result without the secret sauce. I mean, people use all sorts of methods to predict a protein structure, including manual tweaking guided by intuition and insight gained during a reverie or day-dream (à la Kekulé) which is clearly not reproducible. Yet that final model may be publishable, because it may provide new insight and testable predictions.
- msla 3y ago
  My point is that we can, apparently, improve the baseline expectations in the parts of science where this kind of reproducibility is possible. That isn't all science, granted, but it is some science. It isn't a panacea, granted, but it could guard against some forms of misconduct or honest error some of the time. The self-correcting part of science only works when there's something for it to work on, so open data and runnable code ought to improve that self-correction mechanism.
  - eesmith 3y ago
    Understood. But my point is this linked-to essay appears not only to exclude some areas of good science, but to suggest that any topics which cannot be replicated before publication is only worthy of publication in the Journal of Irreproducible Results. I gave examples to highlight why I disagree with author's opinion. Please do not interpret this to mean I do not think improvement is possible.

by msla 3y ago

counters 3y ago

In practice this is happening in many disciplines, for most research, on a daily basis. What _isn't_ happening is that the results of these replications are being independently peer reviewed, because that isn't incentivized. However, when replication fails for whatever reason, it usually leads to insights that themselves lead to stronger scientific work and better publications later on.

eesmith 3y ago

> someone else being able to run that code on that dataset and get the same results. I think when people talk about "replicate" they mean something more than that. The dataset could contain coding errors, and the analysis could contain incorrect formulas and bad modeling. Reproducing a bad analysis, successfully, provide no corrective feedback. I know for one paper I could replicate the paper's results using the paper's own analysis, but I couldn't replicate the paper's results using my analysis. > Would this require labs to give up whatever used to be secret sauce? That's. The. Point. That seems to be a very different Point. Newton famously published results made from using his secret sauce - calculus - by recasting them using more traditional methods. In the extreme cas, I could publish the factors for RSA-1024 without publishing my factorization method. "I prayed to God for the answer and He gave them to me." You can verify that result without the secret sauce. I mean, people use all sorts of methods to predict a protein structure, including manual tweaking guided by intuition and insight gained during a reverie or day-dream (à la Kekulé) which is clearly not reproducible. Yet that final model may be publishable, because it may provide new insight and testable predictions.

msla 3y ago
My point is that we can, apparently, improve the baseline expectations in the parts of science where this kind of reproducibility is possible. That isn't all science, granted, but it is some science. It isn't a panacea, granted, but it could guard against some forms of misconduct or honest error some of the time. The self-correcting part of science only works when there's something for it to work on, so open data and runnable code ought to improve that self-correction mechanism.
- eesmith 3y ago
  Understood. But my point is this linked-to essay appears not only to exclude some areas of good science, but to suggest that any topics which cannot be replicated before publication is only worthy of publication in the Journal of Irreproducible Results. I gave examples to highlight why I disagree with author's opinion. Please do not interpret this to mean I do not think improvement is possible.