3 ms·

at least in some parts of computer science the solution is easy: do not ever publish results without the public source code of all the experiments.

astrange 5y ago
That isn't possible if you're using commercially licensed source from other people, drivers for scientific instruments, lacking copyright assignment for some of it, etc. Same reason many commercial projects can't be open sourced even if the company wanted to.
- enriquto 5y ago
  So people who used proprietary software will not be able to publish. Sounds like a win-win to me!
  - astrange 5y ago
    Your definition of free software is more restrictive than the FSF’s.
    - enriquto 5y ago
      of course i was simplifying... but it seems obvious to me that enforcing automatic reproducibility in peer reviewed publications can only be a good thing in the long run
clavigne 5y ago
I did peer review for a number of scientific papers that include code. Almost every time, I was the only reviewer that even look at the code. In most cases, peer reviewers will just assume that authors claiming the "code is available" means that a) it is reproducible and b) it is actually there. As a counter example, this recent splashy paper https://www.nature.com/articles/s41587-021-00907-6 https://www.nature.com/articles/s41587-021-00907-6 claims the code is available on github, but the github version ( https://github.com/jameswweis/delphi https://github.com/jameswweis/delphi ) contains the actual model only as a Pickle file, and contains no data or featurization. So clearly, the peer reviewers didn't look at it.
- enriquto 5y ago
  that. The main task of the reviewers should be to re-run all the experiments on their own computer and check the results.
  - clavigne 5y ago
    re-running is definitely too much work for most scientific papers, at least in ML and computational sciences were experiments might take 1000s of core-hours or gpu-hours, but that's usually not necessary. In addition, just running the code can spot really bad problems (it doesn't work) but easily miss subtle ones (it works but only for very specific cases). I think it's more important for reviewers to read the source, the same way one would read an experimental protocol and supplementary information, mainly checking for discrepancies between what the paper claims is happening and what is actually being done. In the above example, a reviewer reading the code would have spotted that the model isn't there at all, even though it runs fine.
seoaeu 5y ago
Providing source code is a good thing, but a lot of people confuse re-running experiments with replicating them. If you take the authors' source code and re-run it, then any bugs are going to invalidate your results too. The only way to actually have confidence in the paper's results are to rewrite the software from scratch. In fact, I'd actually go further and question what kinds of errors could possibly be caught be running the same software that the authors did? Any accidental bugs will remain, and any malicious tampering with the experiment data is exceedingly unlikely to be caught even with a careful audit of the code.

astrange 5y ago

That isn't possible if you're using commercially licensed source from other people, drivers for scientific instruments, lacking copyright assignment for some of it, etc. Same reason many commercial projects can't be open sourced even if the company wanted to.

enriquto 5y ago
So people who used proprietary software will not be able to publish. Sounds like a win-win to me!
- astrange 5y ago
  Your definition of free software is more restrictive than the FSF’s.
  - enriquto 5y ago
    of course i was simplifying... but it seems obvious to me that enforcing automatic reproducibility in peer reviewed publications can only be a good thing in the long run

clavigne 5y ago

I did peer review for a number of scientific papers that include code. Almost every time, I was the only reviewer that even look at the code. In most cases, peer reviewers will just assume that authors claiming the "code is available" means that a) it is reproducible and b) it is actually there. As a counter example, this recent splashy paper https://www.nature.com/articles/s41587-021-00907-6 https://www.nature.com/articles/s41587-021-00907-6 claims the code is available on github, but the github version ( https://github.com/jameswweis/delphi https://github.com/jameswweis/delphi ) contains the actual model only as a Pickle file, and contains no data or featurization. So clearly, the peer reviewers didn't look at it.

enriquto 5y ago
that. The main task of the reviewers should be to re-run all the experiments on their own computer and check the results.
- clavigne 5y ago
  re-running is definitely too much work for most scientific papers, at least in ML and computational sciences were experiments might take 1000s of core-hours or gpu-hours, but that's usually not necessary. In addition, just running the code can spot really bad problems (it doesn't work) but easily miss subtle ones (it works but only for very specific cases). I think it's more important for reviewers to read the source, the same way one would read an experimental protocol and supplementary information, mainly checking for discrepancies between what the paper claims is happening and what is actually being done. In the above example, a reviewer reading the code would have spotted that the model isn't there at all, even though it runs fine.

seoaeu 5y ago

Providing source code is a good thing, but a lot of people confuse re-running experiments with replicating them. If you take the authors' source code and re-run it, then any bugs are going to invalidate your results too. The only way to actually have confidence in the paper's results are to rewrite the software from scratch. In fact, I'd actually go further and question what kinds of errors could possibly be caught be running the same software that the authors did? Any accidental bugs will remain, and any malicious tampering with the experiment data is exceedingly unlikely to be caught even with a careful audit of the code.