4 ms·

The simple answer to a question posed in the article "On the other hand one could ask why there isn’t a requirement that published research should be reproducib

The simple answer to a question posed in the article "On the other hand one could ask why there isn’t a requirement that published research should be reproducible using open source tools (or at least tools that are freely available to the public and other researchers)?" is that the obvious immediate outcome of such a requirement is that papers failing that requirement - like the Quviq QuickCheck papers, which seem to have been useful to the author and others - simply would not get published, and the community would lose out on that gift of information.

spencerchubb 2y ago
I think it would be good to have some publishers require reproducibility, and some publishers that don't. Every requirement is exclusionary, and there are always edge cases where a paper can be useful even if it doesn't fulfill a requirement.
- josephg 2y ago
  That’s already the case. Eurosys includes an Artifact Evaluation for submitted papers - which typically includes code and data to allow reviewers to reproduce the work described in the paper. It’s optional, but encouraged. This page lists the artifact evaluation criteria for a handful of conferences: https://sysartifacts.github.io/ https://sysartifacts.github.io/
- ashton314 2y ago
  As an intermediate point in the Require Reproducibility ↔ No Requirement spectrum, strong encouragement to have reproducible artifacts is attractive as well. I just got a paper accepted to ECOOP (European Conference on OO Programming; has lost it's OO-focus and is now a general PL conference) and it's trying something relatively new: artifact evaluations are considered as part of the submission process. Our paper had a reproducible and reusable artifact, and I think that helped our case with the reviewers.
mardifoufs 2y ago
Couldn't it be possible to make the source available for the reviewers only? With whatever is needed to make the code run too. Maybe that's already being done though!
krick 2y ago
Obviously this is not a clear-cut question (one might even call it a political question, and those are notoriously no clear-cut), but this is hardly a valid defence. Because if you consider it valid, you can go however far you like using this statement as a shield. If you abandon reproducibility as a requirement, you don't have to explain anything you don't want to explain. You don't have to provide any data about your samples, no statistical significance tests. A vague abstract claiming such and such result has been achieved would be enough. Heck, the famous Fermat's margin-note on his private copy of Arithmetica is a totally valid research paper then: after all, we wouldn't want to lose out on the precious gift of information that a particular famous mathematician thought he has a concise and elegant proof of a theorem (even though he probably hadn't)! FWIW, my own opinion on that political question is that current standards are way to loose. After all, nobody is forced to publish anything. There is a lot of research happening in the world, that is never published anywhere (e.g. because of proprietary value). It will never go away (unless the world reaches some communist-utopia, but even then it will be doubtful for other reasons). But if one works in academia (and, God forbid, receives grants for his work!) and his stated goal is to move world's scientific knowledge further, it's only fair to ask for him to actually follow that goal, and not just simply pretend following that goal in order to move up that twisted ladder of his academic career.
- vasco 2y ago
  I wholly agree with you, and on the other hand have seen enough researcher code that I know why they don't share it. Usually it's such bad quality, hammered by multiple students to get to some goal, that by the end I was surprised they even trust the output. Publishing would at least make them know from the start this will have to be open sourced and probably even increase the success rate of research by improving coding standards.
Jean-Papoulos 2y ago
Because reproducibility is the cornerstone of the scientific method.
crabbone 2y ago
Papers get published because authors want to increase their "importance index", which is in very direct correlation with how much they will get paid and whether they'll be making a career in academia. Having more requirements towards that end is unlikely to impact the number of papers published. The more severe problem with published papers is mistakes often deliberately overlooked in order to publish as much and as fast as possible. Having papers being easier to verify has a chance of improving this situation... but I wouldn't hold my breath. People are very good at cutting corners.

spencerchubb 2y ago

I think it would be good to have some publishers require reproducibility, and some publishers that don't. Every requirement is exclusionary, and there are always edge cases where a paper can be useful even if it doesn't fulfill a requirement.

josephg 2y ago
That’s already the case. Eurosys includes an Artifact Evaluation for submitted papers - which typically includes code and data to allow reviewers to reproduce the work described in the paper. It’s optional, but encouraged. This page lists the artifact evaluation criteria for a handful of conferences: https://sysartifacts.github.io/ https://sysartifacts.github.io/
ashton314 2y ago
As an intermediate point in the Require Reproducibility ↔ No Requirement spectrum, strong encouragement to have reproducible artifacts is attractive as well. I just got a paper accepted to ECOOP (European Conference on OO Programming; has lost it's OO-focus and is now a general PL conference) and it's trying something relatively new: artifact evaluations are considered as part of the submission process. Our paper had a reproducible and reusable artifact, and I think that helped our case with the reviewers.

mardifoufs 2y ago

Couldn't it be possible to make the source available for the reviewers only? With whatever is needed to make the code run too. Maybe that's already being done though!

krick 2y ago

Obviously this is not a clear-cut question (one might even call it a political question, and those are notoriously no clear-cut), but this is hardly a valid defence. Because if you consider it valid, you can go however far you like using this statement as a shield. If you abandon reproducibility as a requirement, you don't have to explain anything you don't want to explain. You don't have to provide any data about your samples, no statistical significance tests. A vague abstract claiming such and such result has been achieved would be enough. Heck, the famous Fermat's margin-note on his private copy of Arithmetica is a totally valid research paper then: after all, we wouldn't want to lose out on the precious gift of information that a particular famous mathematician thought he has a concise and elegant proof of a theorem (even though he probably hadn't)! FWIW, my own opinion on that political question is that current standards are way to loose. After all, nobody is forced to publish anything. There is a lot of research happening in the world, that is never published anywhere (e.g. because of proprietary value). It will never go away (unless the world reaches some communist-utopia, but even then it will be doubtful for other reasons). But if one works in academia (and, God forbid, receives grants for his work!) and his stated goal is to move world's scientific knowledge further, it's only fair to ask for him to actually follow that goal, and not just simply pretend following that goal in order to move up that twisted ladder of his academic career.

vasco 2y ago
I wholly agree with you, and on the other hand have seen enough researcher code that I know why they don't share it. Usually it's such bad quality, hammered by multiple students to get to some goal, that by the end I was surprised they even trust the output. Publishing would at least make them know from the start this will have to be open sourced and probably even increase the success rate of research by improving coding standards.

Jean-Papoulos 2y ago

Because reproducibility is the cornerstone of the scientific method.

crabbone 2y ago

Papers get published because authors want to increase their "importance index", which is in very direct correlation with how much they will get paid and whether they'll be making a career in academia. Having more requirements towards that end is unlikely to impact the number of papers published. The more severe problem with published papers is mistakes often deliberately overlooked in order to publish as much and as fast as possible. Having papers being easier to verify has a chance of improving this situation... but I wouldn't hold my breath. People are very good at cutting corners.