6 ms·

For a computer scientist, reproducibility means more work that they aren't paid to do. If I ask the Mozilla team to implement new feature X, the response will b

For a computer scientist, reproducibility means more work that they aren't paid to do. If I ask the Mozilla team to implement new feature X, the response will be either (a) point me to a donation link, or (b) We're open source, so why don't you implement the feature yourself? The computer scientist's response is the same.

icebraining 11y ago
It's more like Mozilla writing a new version of Firefox that only runs on their own machines.
jcrites 11y ago
Reproducibility for the computer scientist means including any code written and data collected or relied on in the scientific publication itself. In practice, getting there from here isn't literally zero work, since some actual human action is needed to bundle the code and data, but that effort ought to be negligible overall, especially if we make it a standard part of the scientific process.
- semi-extrinsic 11y ago
  Trust me, it's currently far from zero work to submit code with a research paper. I was recently the corresponding author on a software paper sent to a journal that at least verifies the code compiles and runs and produces expected output. Since the poor person testing the softwares submitted is permanently in the ninth circle of dependency hell, across all platforms and libraries imaginable, it took about fifteen emails back and forth plus an OS reinstall before everything checked out. And they said that wasn't anything extraordinary.
  - jcrites 11y ago
    How about a platform centered around Linux containers (or maybe one of several OS containers or VM images), as the repository image? I'm not saying the work is zero now, but maybe we can get there. If a researcher is developing on a platform where their repository is expressed as a container-like image, then they should be able to publish it for anyone to run exactly as-is. The container repo includes the data, the operating system, and any languages and libraries, with an init system that optionally builds the results.
    - semi-extrinsic 11y ago
      Yes, I think we need to go in this direction. The problem is that the container system is yet another tool for researchers to learn. The first step is to get everyone using VCS and nightly testing. Many are still at the point of clumsily written, old Fortran code that gets emailed around and exists in N different variants. (Not that there is anything wrong with Fortran.) Many are at the point where if you email them a link to a git repo to clone, they're clueless about what to do.
      - Al-Khwarizmi 11y ago
        It would help if Git didn't have such an awful learning curve (and I say this as a git user that already went through it). I know researchers that used Subversion when it was on the rise, but they just abandoned version control altogether when Git became the generally preferred option.
  - eli_gottlieb 11y ago
    Couldn't we all just assume that the reproduction procedure starts from a fresh Ubuntu VM?
  - viraptor 11y ago
    There's a difference though between "just published" and "reviewed, verified and published". We'd gain a lot if anyone simply included the code they were using with the papers. It doesn't matter if it runs and how generic it is. If it's important, it can be fixed by the next user. If it's not, it didn't matter in the first place if it's verified. Of course verification on submission is also a great idea, but we can make it the next step.
morgante 11y ago
In theory, people shouldn't be going into research just to get paid for it. In fact, if they are they're using a pretty ineffective path to get paid.
- KKKKkkkk1 11y ago
  Good point. Research software is always far from production quality and researchers don't get into research to write production software. To publish software, it has to build and run on all reasonable platforms, and users should be able to read, understand and modify the code. All this effort distracts the researcher from doing research. (Unless we redefine research as working on software.)
  - morgante 11y ago
    Nobody is saying your research has to have the quality of industry open source projects. You're just coming up with excuses to continue to hide your code. Merely publishing it at all would be helpful. That way other people can pick it up and modify it to work for them. That's still better than them flying blind and having to completely rewrite your code from scratch. It's really infuriating to see researchers creating this false dichotomy between publishing production-quality software and publishing any code at all.
    - jcrites 11y ago
      Yes, exactly. All I'm asking for is a literal code and data dump. Take a snapshot of the operating environment and publish it. Ideally as a virtual machine or container image, so people don't have to fiddle with dependencies and versions; but if it's just a file system dump of the code that's better than nothing. The folks who are asking for this are NOT asking for: (1) production quality code (2) portable code (3) an open source project based on the work (one way code dump is fine) (4) well commented high quality code (5) any support using the code beyond which would normally be afforded fellow researchers in reproducing one's results. We (me and anyone else who is asking) understand that there may be some circumstances in which not all data can be published due to privacy, or in which code depends on proprietary dependencies that can't be shared. That's fine. Document it if you can as a disclaimer, and people can work on getting access to those private resources if they need to. Anyone funding research should expect this; and publicly funded research should be required to disclose all technical work product that's materially involved in published results.
jordigh 11y ago
I don't think Mozilla Corp takes donations for software development. The Mozilla Foundation does take donations, but it doesn't develop software. Rather, the foundation works on outreach (those cute videos you sometimes see on the Firefox start page). Mozilla is a bit of a weird entity, half non-profit, half very much corporate, yet all code is free.
noedig 11y ago
This is exactly the reason. Academia is a business just like any other. Academics do what they are incentivized to do. I guarantee that if NSF, DOE, NIH and the other US government funding agencies made publishing your code a requirement for funding, and actually checked that you did, you would see everyone start doing it immediately. Right now NSF requires a "data management plan" as part of the proposal, which includes information about sharing, reusing and archiving data and code, but nobody ever checks that you followed the plan after the money is awarded.

icebraining 11y ago

It's more like Mozilla writing a new version of Firefox that only runs on their own machines.

jcrites 11y ago

Reproducibility for the computer scientist means including any code written and data collected or relied on in the scientific publication itself. In practice, getting there from here isn't literally zero work, since some actual human action is needed to bundle the code and data, but that effort ought to be negligible overall, especially if we make it a standard part of the scientific process.

semi-extrinsic 11y ago
Trust me, it's currently far from zero work to submit code with a research paper. I was recently the corresponding author on a software paper sent to a journal that at least verifies the code compiles and runs and produces expected output. Since the poor person testing the softwares submitted is permanently in the ninth circle of dependency hell, across all platforms and libraries imaginable, it took about fifteen emails back and forth plus an OS reinstall before everything checked out. And they said that wasn't anything extraordinary.
- jcrites 11y ago
  How about a platform centered around Linux containers (or maybe one of several OS containers or VM images), as the repository image? I'm not saying the work is zero now, but maybe we can get there. If a researcher is developing on a platform where their repository is expressed as a container-like image, then they should be able to publish it for anyone to run exactly as-is. The container repo includes the data, the operating system, and any languages and libraries, with an init system that optionally builds the results.
  - semi-extrinsic 11y ago
    Yes, I think we need to go in this direction. The problem is that the container system is yet another tool for researchers to learn. The first step is to get everyone using VCS and nightly testing. Many are still at the point of clumsily written, old Fortran code that gets emailed around and exists in N different variants. (Not that there is anything wrong with Fortran.) Many are at the point where if you email them a link to a git repo to clone, they're clueless about what to do.
    - Al-Khwarizmi 11y ago
      It would help if Git didn't have such an awful learning curve (and I say this as a git user that already went through it). I know researchers that used Subversion when it was on the rise, but they just abandoned version control altogether when Git became the generally preferred option.
- eli_gottlieb 11y ago
  Couldn't we all just assume that the reproduction procedure starts from a fresh Ubuntu VM?
- viraptor 11y ago
  There's a difference though between "just published" and "reviewed, verified and published". We'd gain a lot if anyone simply included the code they were using with the papers. It doesn't matter if it runs and how generic it is. If it's important, it can be fixed by the next user. If it's not, it didn't matter in the first place if it's verified. Of course verification on submission is also a great idea, but we can make it the next step.

morgante 11y ago

In theory, people shouldn't be going into research just to get paid for it. In fact, if they are they're using a pretty ineffective path to get paid.

KKKKkkkk1 11y ago
Good point. Research software is always far from production quality and researchers don't get into research to write production software. To publish software, it has to build and run on all reasonable platforms, and users should be able to read, understand and modify the code. All this effort distracts the researcher from doing research. (Unless we redefine research as working on software.)
- morgante 11y ago
  Nobody is saying your research has to have the quality of industry open source projects. You're just coming up with excuses to continue to hide your code. Merely publishing it at all would be helpful. That way other people can pick it up and modify it to work for them. That's still better than them flying blind and having to completely rewrite your code from scratch. It's really infuriating to see researchers creating this false dichotomy between publishing production-quality software and publishing any code at all.
  - jcrites 11y ago
    Yes, exactly. All I'm asking for is a literal code and data dump. Take a snapshot of the operating environment and publish it. Ideally as a virtual machine or container image, so people don't have to fiddle with dependencies and versions; but if it's just a file system dump of the code that's better than nothing. The folks who are asking for this are NOT asking for: (1) production quality code (2) portable code (3) an open source project based on the work (one way code dump is fine) (4) well commented high quality code (5) any support using the code beyond which would normally be afforded fellow researchers in reproducing one's results. We (me and anyone else who is asking) understand that there may be some circumstances in which not all data can be published due to privacy, or in which code depends on proprietary dependencies that can't be shared. That's fine. Document it if you can as a disclaimer, and people can work on getting access to those private resources if they need to. Anyone funding research should expect this; and publicly funded research should be required to disclose all technical work product that's materially involved in published results.

jordigh 11y ago

I don't think Mozilla Corp takes donations for software development. The Mozilla Foundation does take donations, but it doesn't develop software. Rather, the foundation works on outreach (those cute videos you sometimes see on the Firefox start page). Mozilla is a bit of a weird entity, half non-profit, half very much corporate, yet all code is free.

noedig 11y ago

This is exactly the reason. Academia is a business just like any other. Academics do what they are incentivized to do. I guarantee that if NSF, DOE, NIH and the other US government funding agencies made publishing your code a requirement for funding, and actually checked that you did, you would see everyone start doing it immediately. Right now NSF requires a "data management plan" as part of the proposal, which includes information about sharing, reusing and archiving data and code, but nobody ever checks that you followed the plan after the money is awarded.