6 ms·

Reproducible? Or deterministic? There's certainly benefits to being able to pull down research source code, and bug checking it. That's how programmers check c

Reproducible? Or deterministic? There's certainly benefits to being able to pull down research source code, and bug checking it. That's how programmers check code: tests and audits. However I think reproducing research is more often then not done "from scratch", taking a new sample, treating it, checking results. "independent verification". Re-using source code saves time, but I would argue not being able to shouldn't threaten reproducibility.

davnn 6y ago
> Re-using source code saves time, but I would argue not being able to shouldn't threaten reproducibility. More often than not it‘s not clear from a paper what exactly the authors did to a achieve a specific result. Being able to exactly reproduce what previous authors did should improve reproducibility; also for new samples.
- asdff 6y ago
  It's also standard fare for typical lab work. A good paper's methods section would contain enough detail for you to go into your lab and repeat the experiment yourself, even down to the catalog number for the reagents to order from the lab supplier. Code should be no different, that's why it's encouraged that authors submit all code used in analysis and generation of figures.
  - davnn 6y ago
    The fun thing is that there are approaches that want to go beyond this kind of methodological description of a scientific process to code [0, 1]. In general I would say that the more we can remove the human aspect and inherent ambiguity of science, the better for reproducibility. See [2] for a couple of examples. [0] https://www.emeraldcloudlab.com/ https://www.emeraldcloudlab.com/ [1] https://nextjournal.com/ https://nextjournal.com/ [2] https://www.youtube.com/watch?v=L1UgdoP2aeg https://www.youtube.com/watch?v=L1UgdoP2aeg
coliveira 6y ago
That's something that eludes software people: reproducibility in science is the ability to create independent tests. Making software available, while useful, does very little for reproducibility from the scientific point of view.
- petters 6y ago
  It's true that "reproduce" can mean different things in software and science. But having the code available to "reproduce" any plots in a paper should be a requirement for publication, imo. It certainly is the case for my papers.
- f6v 6y ago
  There’re fields of science, like computational biology, where it’s all about the code. I wish the methods section was always 100% unambiguous, but it’s not the case. And nowadays the computational pipelines have to support analysis of up to terabytes of data. You can imagine how many dependencies such pipelines have. Sometimes I have trouble installing the software even when package manager such as anaconda is used.
- bonoboTP 6y ago
  It's a minimum standard, though. Of course the goal is reproducibility from a broader point of view, but that's not an excuse to do research in a one-off way where nobody is able to show how to get those numbers again, after a year or so from publication. The coding standards are often abysmally, unexpectedly terrible. Often not even the help of the original authors is enough to be able to produce the same figures from a paper because things and settings and commands get forgotten. Some part of the analysis was done in one language, another part in Excel. Some of the code has now disappeared. Some of the libraries are no longer working. Some people left and their academic storage space was wiped and therefore the intermediate steps and results or notes are deleted. You wouldn't believe it. Once a paper is published researchers are not really incentivized to document things or maintain the materials. They got the publication, they put it on their CV. On to the next project! No time to waste on work that's already completed. New work leads to new publications, messing around with the old code for the sake of a potential later person interested in it is a waste from the point of view of a researcher, career wise. Also most papers are never attempted to be reproduced ever.
jdale27 6y ago
Ideally research does get reproduced from scratch; I think what people usually mean when they talk about the replication/reproducibility crisis in science is not being able to reproduce an experiment with new samples, independent data analysis, etc. However, if you can't even reproduce an analysis with the authors' own data and code, that's a red flag before you even get to the starting line. Ensuring that level of reproducibility is, I think, an essential ingredient to enabling the stronger form of reproducibility. Personally, I made the mistake during my graduate career of trying to reimplement an analysis using a certain rather complicated ML algorithm, from scratch, in a different language than the original authors had used. After struggling mightily to get it to work, I finally bothered to try to get their own code working. (I had been hesitant to do so because I wasn't proficient in the language they used, and it wasn't even clear they had released all the necessary code, aside from the core algorithm.) Once I did that, I discovered that I couldn't even get their own code working on their own data, and gave up. This was researched published in Science by a group from a top-tier research university. (I don't fully blame the authors, it may well have been my own incompetence that was the issue. But it just serves as yet another illustration of how pervasive and disregarded the reproducibility issue was for a long while.)
geomark 6y ago
Back when I studied this stuff there was a distinction between reproducible (rerun analysis on the data from the original experiment and see if you get the same results - if not then there is an error in the analysis) and replicable (redo the entire experiment by taking new data and running the analisys).
bonoboTP 6y ago
This is why people are starting to make a difference between terms: repeatability, reproducibility, replicability. > You give me your code and enough information for me to produce and identical environment or (even better) your code is insenstive the environment, then your research is Repeatable. > If you describe your study sufficiently well that I can re-implement your study from scratch, without looking at your code and still get the same answer, then it is Reproducible > If I can arrive at the same conclusions as you, just from a description of its aims, then it is Replicable. From https://academia.stackexchange.com/a/118518/15198 https://academia.stackexchange.com/a/118518/15198

davnn 6y ago

> Re-using source code saves time, but I would argue not being able to shouldn't threaten reproducibility. More often than not it‘s not clear from a paper what exactly the authors did to a achieve a specific result. Being able to exactly reproduce what previous authors did should improve reproducibility; also for new samples.

asdff 6y ago
It's also standard fare for typical lab work. A good paper's methods section would contain enough detail for you to go into your lab and repeat the experiment yourself, even down to the catalog number for the reagents to order from the lab supplier. Code should be no different, that's why it's encouraged that authors submit all code used in analysis and generation of figures.
- davnn 6y ago
  The fun thing is that there are approaches that want to go beyond this kind of methodological description of a scientific process to code [0, 1]. In general I would say that the more we can remove the human aspect and inherent ambiguity of science, the better for reproducibility. See [2] for a couple of examples. [0] https://www.emeraldcloudlab.com/ https://www.emeraldcloudlab.com/ [1] https://nextjournal.com/ https://nextjournal.com/ [2] https://www.youtube.com/watch?v=L1UgdoP2aeg https://www.youtube.com/watch?v=L1UgdoP2aeg

coliveira 6y ago

That's something that eludes software people: reproducibility in science is the ability to create independent tests. Making software available, while useful, does very little for reproducibility from the scientific point of view.

petters 6y ago
It's true that "reproduce" can mean different things in software and science. But having the code available to "reproduce" any plots in a paper should be a requirement for publication, imo. It certainly is the case for my papers.
f6v 6y ago
There’re fields of science, like computational biology, where it’s all about the code. I wish the methods section was always 100% unambiguous, but it’s not the case. And nowadays the computational pipelines have to support analysis of up to terabytes of data. You can imagine how many dependencies such pipelines have. Sometimes I have trouble installing the software even when package manager such as anaconda is used.
bonoboTP 6y ago
It's a minimum standard, though. Of course the goal is reproducibility from a broader point of view, but that's not an excuse to do research in a one-off way where nobody is able to show how to get those numbers again, after a year or so from publication. The coding standards are often abysmally, unexpectedly terrible. Often not even the help of the original authors is enough to be able to produce the same figures from a paper because things and settings and commands get forgotten. Some part of the analysis was done in one language, another part in Excel. Some of the code has now disappeared. Some of the libraries are no longer working. Some people left and their academic storage space was wiped and therefore the intermediate steps and results or notes are deleted. You wouldn't believe it. Once a paper is published researchers are not really incentivized to document things or maintain the materials. They got the publication, they put it on their CV. On to the next project! No time to waste on work that's already completed. New work leads to new publications, messing around with the old code for the sake of a potential later person interested in it is a waste from the point of view of a researcher, career wise. Also most papers are never attempted to be reproduced ever.

jdale27 6y ago

Ideally research does get reproduced from scratch; I think what people usually mean when they talk about the replication/reproducibility crisis in science is not being able to reproduce an experiment with new samples, independent data analysis, etc. However, if you can't even reproduce an analysis with the authors' own data and code, that's a red flag before you even get to the starting line. Ensuring that level of reproducibility is, I think, an essential ingredient to enabling the stronger form of reproducibility. Personally, I made the mistake during my graduate career of trying to reimplement an analysis using a certain rather complicated ML algorithm, from scratch, in a different language than the original authors had used. After struggling mightily to get it to work, I finally bothered to try to get their own code working. (I had been hesitant to do so because I wasn't proficient in the language they used, and it wasn't even clear they had released all the necessary code, aside from the core algorithm.) Once I did that, I discovered that I couldn't even get their own code working on their own data, and gave up. This was researched published in Science by a group from a top-tier research university. (I don't fully blame the authors, it may well have been my own incompetence that was the issue. But it just serves as yet another illustration of how pervasive and disregarded the reproducibility issue was for a long while.)

geomark 6y ago

Back when I studied this stuff there was a distinction between reproducible (rerun analysis on the data from the original experiment and see if you get the same results - if not then there is an error in the analysis) and replicable (redo the entire experiment by taking new data and running the analisys).

bonoboTP 6y ago

This is why people are starting to make a difference between terms: repeatability, reproducibility, replicability. > You give me your code and enough information for me to produce and identical environment or (even better) your code is insenstive the environment, then your research is Repeatable. > If you describe your study sufficiently well that I can re-implement your study from scratch, without looking at your code and still get the same answer, then it is Reproducible > If I can arrive at the same conclusions as you, just from a description of its aims, then it is Replicable. From https://academia.stackexchange.com/a/118518/15198 https://academia.stackexchange.com/a/118518/15198