5 ms·

Don't forget code! "we then used our in-house untested code, developed by an undergrad we scrapped from Biology, to normalize the results. The code is comfortab

by avip 9y ago

Don't forget code! "we then used our in-house untested code, developed by an undergrad we scrapped from Biology, to normalize the results. The code is comfortably not attached"

kazinator 9y ago
And that's just the CS papers.
- ziotom78 9y ago
  Not entirely true: I am an astrophysicist, and I can say that this practice would be extremely useful for my discipline as well. In cosmology (my field) we use codes that are often extremely complex, yet they are not usually released together with the papers based on their outputs. This puzzles me, as last year I attended a conference on informatics and astrophysics (ADASS), and one of the talks showed that releasing your codes as open source increases a lot the chance of your papers being cited (expecially if you advertise your code using services like ASCL [1]). The only reason I can think of this unwillingness to publish codes is the fact that these codes written by physicists are often extremely unreadable: very long Fortran routines with awkward variable names, no comments, no test cases, no documentation… Once you get a result from your own software, you get more satisfaction if you publish a paper with the results than if you polish and publish the code. IMHO this is bad science, but it is difficult to change this way of working: cosmology is today done by large collaborations, not by individuals, and if you propose to make the codebase developed by your team public, this idea is usually not welcomed by your co-authors. [1] http://ascl.net/ http://ascl.net/
eli 9y ago
To be fair: "All computer codes involved in the creation or analysis of data must also be available to any reader of Science." And "Upon publication, Nature Journals consider it best practice to release custom computer code in a way that allows readers to repeat the published results."
- jcrites 9y ago
  Could we take it one step further? I'd love to see a publication format that bundles the code, the data, and maybe even the paper together into a single executable bundle for distribution. Imagine that the bundle starts with a manifest file that describes the data and the code, pointing to an authoritative, versioned copy. The user can "run" the manifest, which will download and install the data and any necessary dependencies. (If you're thinking about this from a containerized point of view, the code dependencies and data may be modeled as a container or virtual machine image.) The manifest is like a build file that executes the code passing the data as input, in a clear and reproducible way. Anyone running the manifest will get the same results as the researcher did. Because the entirety of the execution environment is versioned and captured by the manifest, including all of the system software, the results are reproducible from the code and data phase onward with zero effort. For bonus points, the manifest would also "build" the paper (e.g. if it's TeX), and would substitute the results of the code execution directly into the paper, i.e. graphs and numbers. You could conceptualize the paper as something like a notebook, where its values can change dynamically according with the data you provide it.
  - jackcosgrove 9y ago
    Sounds like a Jupyter Notebook.
  - godelmachine 9y ago
    Would all this be possible using git? In that case, even normal people (non-professors , non-R&D engineers) who read research papers to quench their curiosity, would have to know to use git.
    - evincarofautumn 9y ago
      You could use an open protocol based on git, Docker, &c. internally, and build some interface friendly to non-programmers on top.
      - maxander 9y ago

by avip 9y ago

Don't forget code! "we then used our in-house untested code, developed by an undergrad we scrapped from Biology, to normalize the results. The code is comfortably not attached"

kazinator 9y ago

And that's just the CS papers.

ziotom78 9y ago
Not entirely true: I am an astrophysicist, and I can say that this practice would be extremely useful for my discipline as well. In cosmology (my field) we use codes that are often extremely complex, yet they are not usually released together with the papers based on their outputs. This puzzles me, as last year I attended a conference on informatics and astrophysics (ADASS), and one of the talks showed that releasing your codes as open source increases a lot the chance of your papers being cited (expecially if you advertise your code using services like ASCL [1]). The only reason I can think of this unwillingness to publish codes is the fact that these codes written by physicists are often extremely unreadable: very long Fortran routines with awkward variable names, no comments, no test cases, no documentation… Once you get a result from your own software, you get more satisfaction if you publish a paper with the results than if you polish and publish the code. IMHO this is bad science, but it is difficult to change this way of working: cosmology is today done by large collaborations, not by individuals, and if you propose to make the codebase developed by your team public, this idea is usually not welcomed by your co-authors. [1] http://ascl.net/ http://ascl.net/

eli 9y ago

To be fair: "All computer codes involved in the creation or analysis of data must also be available to any reader of Science." And "Upon publication, Nature Journals consider it best practice to release custom computer code in a way that allows readers to repeat the published results."

jcrites 9y ago
Could we take it one step further? I'd love to see a publication format that bundles the code, the data, and maybe even the paper together into a single executable bundle for distribution. Imagine that the bundle starts with a manifest file that describes the data and the code, pointing to an authoritative, versioned copy. The user can "run" the manifest, which will download and install the data and any necessary dependencies. (If you're thinking about this from a containerized point of view, the code dependencies and data may be modeled as a container or virtual machine image.) The manifest is like a build file that executes the code passing the data as input, in a clear and reproducible way. Anyone running the manifest will get the same results as the researcher did. Because the entirety of the execution environment is versioned and captured by the manifest, including all of the system software, the results are reproducible from the code and data phase onward with zero effort. For bonus points, the manifest would also "build" the paper (e.g. if it's TeX), and would substitute the results of the code execution directly into the paper, i.e. graphs and numbers. You could conceptualize the paper as something like a notebook, where its values can change dynamically according with the data you provide it.
- jackcosgrove 9y ago
  Sounds like a Jupyter Notebook.
- godelmachine 9y ago
  Would all this be possible using git? In that case, even normal people (non-professors , non-R&D engineers) who read research papers to quench their curiosity, would have to know to use git.
  - evincarofautumn 9y ago
    You could use an open protocol based on git, Docker, &c. internally, and build some interface friendly to non-programmers on top.
    - maxander 9y ago