6 ms·

The solution is obvious - provide a github repository with all the code for the research. Provide clear ways to reproduce all the results of the research from r

The solution is obvious - provide a github repository with all the code for the research. Provide clear ways to reproduce all the results of the research from raw data by running build. Bonus points: Hirschware as software license.

Natanael_L 11y ago
Maybe even a container image / VM to ensure there's no subtle enviroment differences that alter the results. Even better with reproducible builds specifically so you can confirm it was created from the published source.
- guard-of-terra 11y ago
  IMO that's an overkill. It's sufficient that the result is reproduced once. If slightly different environment would produce enormously different results, that's interesting by itself. The goal is that you can continue research based on previous results, not that you can reproduce exact results in an obtuse VM.
mattlutze 11y ago
Isn't that what the article describes? The core of the article is first recommendations on writing scripts to disambiguate cleanup tasks in data and organization/records management of files. It's then recommendations to use open source languages for scripting / analysis and for using git to maintain it all. The trouble you run into is in the phrase "Provide clear ways to reproduce all the results." There is "clear ways to," then there is "an exact method to." Academic life is cutthroat and research tenure and continued employment rely on showing that other peers are using their research. It's common practice to write an article in such a way that it looks "clear" but requires consultation with the researcher to understand precisely. Ambiguity is part and parcel. The tough need here is core journals in each field being more rigorous on the sussing out of not-quite-good-enough methods statements to force researchers to be more open.
amelius 11y ago
But if somebody would read the publication in, say, 20 years, then the whole platform on which the computation has to run might have become obsolete. Suddenly, you have to look for emulators. Or even for emulators running emulators.
- guard-of-terra 11y ago
  With some effort you could make it work, perhaps make a pull request. Programming languages will likely still be around. Using open formats and tools would likely help.
  - amelius 11y ago
    Well to give an example: good luck finding an Ada compiler that compiles to PDP-11, and then good luck getting it to run :) And this is even an example of a combination which once was considered mainstream.
  - rspeer 11y ago
    You won't succeed at making a "pull request" in 2035 against a piece of code written in 2015. The person who wrote the code has another job now. The programming language it's written in is unsupported and unsafe to run. The site it's hosted on was sold to a holding company and then taken down. The entire concept of a "pull request" has been replaced by some new idea that's as foreign to us as a pull request would be to developers in 1995.
    - sopooneo 11y ago
      Right. I look down the road when the drive isn't to migrate to Python 3, but from it, to PyGo 1.4. And damn-it, what is this PoGo thing? We had PIP! PIP worked! Why are this tri-ennials continually reinventing stuff we had completing worked out by the early 20's?!
cozzyd 11y ago
git? Maybe in 10 years. Some of us just moved to svn from cvs... (and many are still using cvs).

Natanael_L 11y ago

Maybe even a container image / VM to ensure there's no subtle enviroment differences that alter the results. Even better with reproducible builds specifically so you can confirm it was created from the published source.

guard-of-terra 11y ago
IMO that's an overkill. It's sufficient that the result is reproduced once. If slightly different environment would produce enormously different results, that's interesting by itself. The goal is that you can continue research based on previous results, not that you can reproduce exact results in an obtuse VM.

mattlutze 11y ago

Isn't that what the article describes? The core of the article is first recommendations on writing scripts to disambiguate cleanup tasks in data and organization/records management of files. It's then recommendations to use open source languages for scripting / analysis and for using git to maintain it all. The trouble you run into is in the phrase "Provide clear ways to reproduce all the results." There is "clear ways to," then there is "an exact method to." Academic life is cutthroat and research tenure and continued employment rely on showing that other peers are using their research. It's common practice to write an article in such a way that it looks "clear" but requires consultation with the researcher to understand precisely. Ambiguity is part and parcel. The tough need here is core journals in each field being more rigorous on the sussing out of not-quite-good-enough methods statements to force researchers to be more open.

amelius 11y ago

But if somebody would read the publication in, say, 20 years, then the whole platform on which the computation has to run might have become obsolete. Suddenly, you have to look for emulators. Or even for emulators running emulators.

guard-of-terra 11y ago
With some effort you could make it work, perhaps make a pull request. Programming languages will likely still be around. Using open formats and tools would likely help.
- amelius 11y ago
  Well to give an example: good luck finding an Ada compiler that compiles to PDP-11, and then good luck getting it to run :) And this is even an example of a combination which once was considered mainstream.
- rspeer 11y ago
  You won't succeed at making a "pull request" in 2035 against a piece of code written in 2015. The person who wrote the code has another job now. The programming language it's written in is unsupported and unsafe to run. The site it's hosted on was sold to a holding company and then taken down. The entire concept of a "pull request" has been replaced by some new idea that's as foreign to us as a pull request would be to developers in 1995.
  - sopooneo 11y ago
    Right. I look down the road when the drive isn't to migrate to Python 3, but from it, to PyGo 1.4. And damn-it, what is this PoGo thing? We had PIP! PIP worked! Why are this tri-ennials continually reinventing stuff we had completing worked out by the early 20's?!

cozzyd 11y ago

git? Maybe in 10 years. Some of us just moved to svn from cvs... (and many are still using cvs).