3 ms·

> the real test of a paper should be the ability to reproduce its findings in the real world. ... > What if all the experiments in the paper are too complicat

> the real test of a paper should be the ability to reproduce its findings in the real world. ... > What if all the experiments in the paper are too complicated to replicate? Then you can submit to [the Journal of Irreproducible Results]. Observational science is still a branch of science even if it's difficult or impossible to replicate. Consider the first photographs of a live giant squid in its natural habitat, published in 2005 at https://royalsocietypublishing.org/doi/10.1098/rspb.2005.3158 https://royalsocietypublishing.org/doi/10.1098/rspb.2005.315... . Who seriously thinks this shouldn't have been published until someone else had been able to replicate the result? Who thinks the results of a drug trial can't be published until they are replicated? How does one replicate "A stellar occultation by (486958) 2014 MU69: results from the 2017 July 17 portable telescope campaign" at https://ui.adsabs.harvard.edu/abs/2017DPS....4950403Z/abstract https://ui.adsabs.harvard.edu/abs/2017DPS....4950403Z/abstra... which required the precise alignment of a star, the trans-Neptunian object 486958 Arrokoth, and a region in Argentina? Or replicate the results of the flyby of Pluto, or flying a helicopter on Mars? Here's a paper I learned about from "In The Pipeline"; "Insights from a laboratory fire" at https://www.nature.com/articles/s41557-023-01254-6 https://www.nature.com/articles/s41557-023-01254-6 . """Fires are relatively common yet underreported occurrences in chemical laboratories, but their consequences can be devastating. Here we describe our first-hand experience of a savage laboratory fire, highlighting the detrimental effects that it had on the research group and the lessons learned.""" How would peer replication be relevant?

msla 3y ago
With some of the things, but admittedly not most of the things you mentioned, there's a dataset (somewhere) and some code run on that dataset (somewhere) and replication would mean someone else being able to run that code on that dataset and get the same results. Would this require labs to improve their software environments and learn some new tools? Would this require labs to give up whatever used to be secret sauce? That's. The. Point.
- counters 3y ago
  In practice this is happening in many disciplines, for most research, on a daily basis. What _isn't_ happening is that the results of these replications are being independently peer reviewed, because that isn't incentivized. However, when replication fails for whatever reason, it usually leads to insights that themselves lead to stronger scientific work and better publications later on.
- eesmith 3y ago
  > someone else being able to run that code on that dataset and get the same results. I think when people talk about "replicate" they mean something more than that. The dataset could contain coding errors, and the analysis could contain incorrect formulas and bad modeling. Reproducing a bad analysis, successfully, provide no corrective feedback. I know for one paper I could replicate the paper's results using the paper's own analysis, but I couldn't replicate the paper's results using my analysis. > Would this require labs to give up whatever used to be secret sauce? That's. The. Point. That seems to be a very different Point. Newton famously published results made from using his secret sauce - calculus - by recasting them using more traditional methods. In the extreme cas, I could publish the factors for RSA-1024 without publishing my factorization method. "I prayed to God for the answer and He gave them to me." You can verify that result without the secret sauce. I mean, people use all sorts of methods to predict a protein structure, including manual tweaking guided by intuition and insight gained during a reverie or day-dream (à la Kekulé) which is clearly not reproducible. Yet that final model may be publishable, because it may provide new insight and testable predictions.
  - msla 3y ago
    My point is that we can, apparently, improve the baseline expectations in the parts of science where this kind of reproducibility is possible. That isn't all science, granted, but it is some science. It isn't a panacea, granted, but it could guard against some forms of misconduct or honest error some of the time. The self-correcting part of science only works when there's something for it to work on, so open data and runnable code ought to improve that self-correction mechanism.
    - eesmith 3y ago
      Understood. But my point is this linked-to essay appears not only to exclude some areas of good science, but to suggest that any topics which cannot be replicated before publication is only worthy of publication in the Journal of Irreproducible Results. I gave examples to highlight why I disagree with author's opinion. Please do not interpret this to mean I do not think improvement is possible.
phpisthebest 3y ago
I think in some of those cases you have conclusions drawn from raw data that could be replicated or reviewed. For example many teams use the same raw data from Large Colliders, or JWT, or other large science projects to reach competiting conclusions. Yes in a perfect world we would also replicate the data collection but we do not live in a perfect world Same is true for Drug Trials, there is always a battle over getting the raw data from drug trails as the companies claim that data is trade secret, so independent verification of drug trails is very expensive but if the FDA required not just the release of redacted conclusions and supporting redacted data but 100% of all data gathered it would be alot better IMO For example the FDA says it will take decades to release the raw data from the COVID Vaccine trials.. Why... and that is after being forced to do so via a law suit.
- eesmith 3y ago
  > For example many teams use the same raw data from Large Colliders, or JWT, or other large science projects to reach competiting conclusions. Yes, but why must the first team wait until the second is finished before publishing? What if you are the only person in the world with expertise in the fossil record of an obscure branch of snails? You spend 10 years developing a paper knowing that the next person with the right training to replicate the work might not even be born yet. Other paleontologists might not be able to replicate the work, but still tell if it's publishable - that's what they do now, yes? > but we do not live in a perfect world Alternatively, we don't live in a perfect world which is why we have the current system instead of requiring replication first. Since the same logic works for both cases, I don't think it's persuasive logic. > the FDA says it will take decades Well, that's a tangent. The FDA is charged with protecting and promoting public health, not improving the state of scholarly literature. And the FDA is only one of many public health organizations which carried out COVID vaccine trials.
kergonath 3y ago
> Who seriously thinks this shouldn't have been published until someone else had been able to replicate the result? Nobody, obviously. You cannot reproduce a result that hasn’t been published, so no new phenomenon is replicated the moment it is first published. The problem is not the publication of new discoveries, it’s the lack of incentives to confirm them once they’ve been published. In your example, new observations of giant squids are still massively valuable even if not that novel anymore. So new observations should be encouraged (as I am sure they are). > Or replicate the results of the flyby of Pluto, or flying a helicopter on Mars? Well, we should launch another probe anyway. And I am fairly confident we’ll have many instances of aircrafts in Mars’ atmosphere and more data than we’ll know what to do with it. We can also simulate the hell out of it. We’ll point spectrometers and a whole bunch of instruments towards Pluto. These are not really good examples of unreproducible observations. Besides, in such cases robustness can be improved by different teams performing their own analyses separately, even if the data comes from the same experimental setup. It’s not all black or white. Observations are on a spectrum, some of them being much more reliable than others and replication is one aspect of it. > How would peer replication be relevant? How would you know which aspects of the observed phenomena come from particularities of this specific lab? You need more than one instance. You need some kind of statistical and factor analyses. Replication in this instance would not mean setting actual labs on fire on purpose. It’s exactly like studying car crashes: nobody is going to kill people on purpose, but it is still important to study them so we regularly have new papers on the subject based on events that happened anyway, each one confirming or disproving previous observations.
- eesmith 3y ago
  > Nobody, obviously. You cannot reproduce a result that hasn’t been published, .. The problem is not the publication of new discoveries, it’s the lack of incentives to confirm them once they’ve been published. Your comment concerns post-publication peer-replication, yes? If so, it's a different topic. The linked-to essay specifically proposes: ""Instead of sending out a manuscript to anonymous referees to read and review, preprints should be sent to other labs to actually replicate the findings. Once the key findings are replicated, the manuscript would be accepted and published."" That's pre-publication peer-replication, and my comment was only meant to be interpreted in that light.

msla 3y ago

With some of the things, but admittedly not most of the things you mentioned, there's a dataset (somewhere) and some code run on that dataset (somewhere) and replication would mean someone else being able to run that code on that dataset and get the same results. Would this require labs to improve their software environments and learn some new tools? Would this require labs to give up whatever used to be secret sauce? That's. The. Point.

counters 3y ago
In practice this is happening in many disciplines, for most research, on a daily basis. What _isn't_ happening is that the results of these replications are being independently peer reviewed, because that isn't incentivized. However, when replication fails for whatever reason, it usually leads to insights that themselves lead to stronger scientific work and better publications later on.
eesmith 3y ago
> someone else being able to run that code on that dataset and get the same results. I think when people talk about "replicate" they mean something more than that. The dataset could contain coding errors, and the analysis could contain incorrect formulas and bad modeling. Reproducing a bad analysis, successfully, provide no corrective feedback. I know for one paper I could replicate the paper's results using the paper's own analysis, but I couldn't replicate the paper's results using my analysis. > Would this require labs to give up whatever used to be secret sauce? That's. The. Point. That seems to be a very different Point. Newton famously published results made from using his secret sauce - calculus - by recasting them using more traditional methods. In the extreme cas, I could publish the factors for RSA-1024 without publishing my factorization method. "I prayed to God for the answer and He gave them to me." You can verify that result without the secret sauce. I mean, people use all sorts of methods to predict a protein structure, including manual tweaking guided by intuition and insight gained during a reverie or day-dream (à la Kekulé) which is clearly not reproducible. Yet that final model may be publishable, because it may provide new insight and testable predictions.
- msla 3y ago
  My point is that we can, apparently, improve the baseline expectations in the parts of science where this kind of reproducibility is possible. That isn't all science, granted, but it is some science. It isn't a panacea, granted, but it could guard against some forms of misconduct or honest error some of the time. The self-correcting part of science only works when there's something for it to work on, so open data and runnable code ought to improve that self-correction mechanism.
  - eesmith 3y ago
    Understood. But my point is this linked-to essay appears not only to exclude some areas of good science, but to suggest that any topics which cannot be replicated before publication is only worthy of publication in the Journal of Irreproducible Results. I gave examples to highlight why I disagree with author's opinion. Please do not interpret this to mean I do not think improvement is possible.

phpisthebest 3y ago

I think in some of those cases you have conclusions drawn from raw data that could be replicated or reviewed. For example many teams use the same raw data from Large Colliders, or JWT, or other large science projects to reach competiting conclusions. Yes in a perfect world we would also replicate the data collection but we do not live in a perfect world Same is true for Drug Trials, there is always a battle over getting the raw data from drug trails as the companies claim that data is trade secret, so independent verification of drug trails is very expensive but if the FDA required not just the release of redacted conclusions and supporting redacted data but 100% of all data gathered it would be alot better IMO For example the FDA says it will take decades to release the raw data from the COVID Vaccine trials.. Why... and that is after being forced to do so via a law suit.

eesmith 3y ago
> For example many teams use the same raw data from Large Colliders, or JWT, or other large science projects to reach competiting conclusions. Yes, but why must the first team wait until the second is finished before publishing? What if you are the only person in the world with expertise in the fossil record of an obscure branch of snails? You spend 10 years developing a paper knowing that the next person with the right training to replicate the work might not even be born yet. Other paleontologists might not be able to replicate the work, but still tell if it's publishable - that's what they do now, yes? > but we do not live in a perfect world Alternatively, we don't live in a perfect world which is why we have the current system instead of requiring replication first. Since the same logic works for both cases, I don't think it's persuasive logic. > the FDA says it will take decades Well, that's a tangent. The FDA is charged with protecting and promoting public health, not improving the state of scholarly literature. And the FDA is only one of many public health organizations which carried out COVID vaccine trials.

kergonath 3y ago

> Who seriously thinks this shouldn't have been published until someone else had been able to replicate the result? Nobody, obviously. You cannot reproduce a result that hasn’t been published, so no new phenomenon is replicated the moment it is first published. The problem is not the publication of new discoveries, it’s the lack of incentives to confirm them once they’ve been published. In your example, new observations of giant squids are still massively valuable even if not that novel anymore. So new observations should be encouraged (as I am sure they are). > Or replicate the results of the flyby of Pluto, or flying a helicopter on Mars? Well, we should launch another probe anyway. And I am fairly confident we’ll have many instances of aircrafts in Mars’ atmosphere and more data than we’ll know what to do with it. We can also simulate the hell out of it. We’ll point spectrometers and a whole bunch of instruments towards Pluto. These are not really good examples of unreproducible observations. Besides, in such cases robustness can be improved by different teams performing their own analyses separately, even if the data comes from the same experimental setup. It’s not all black or white. Observations are on a spectrum, some of them being much more reliable than others and replication is one aspect of it. > How would peer replication be relevant? How would you know which aspects of the observed phenomena come from particularities of this specific lab? You need more than one instance. You need some kind of statistical and factor analyses. Replication in this instance would not mean setting actual labs on fire on purpose. It’s exactly like studying car crashes: nobody is going to kill people on purpose, but it is still important to study them so we regularly have new papers on the subject based on events that happened anyway, each one confirming or disproving previous observations.

eesmith 3y ago
> Nobody, obviously. You cannot reproduce a result that hasn’t been published, .. The problem is not the publication of new discoveries, it’s the lack of incentives to confirm them once they’ve been published. Your comment concerns post-publication peer-replication, yes? If so, it's a different topic. The linked-to essay specifically proposes: ""Instead of sending out a manuscript to anonymous referees to read and review, preprints should be sent to other labs to actually replicate the findings. Once the key findings are replicated, the manuscript would be accepted and published."" That's pre-publication peer-replication, and my comment was only meant to be interpreted in that light.