4 ms·

In all my papers the results were produced on multiple days (spanning months), with multiple versions of the code, and they are computationally too expensive to

by jhrmnn 6y ago

In all my papers the results were produced on multiple days (spanning months), with multiple versions of the code, and they are computationally too expensive to reproduce with the final version of the code. I'm trying to keep track of all the used versions, but given that there is no automated framework for this (is there?) and research involves lots of experiments, it's never perfect. Given this context, any ideas how to do it better?

xen0 6y ago
My first thought: Demand the journals provide hosting for a code repo that is part of your paper. For every numerical result, specify the version (e.g. a git tag) used to generate your result. And if that means scientists need to learn about version control, well... they should if they're writing code.
- chriswarbo 6y ago
  I agree, except that AFAIK "tags" in git are not fixed, they can be deleted and re-created to point at a different commit. Hence I prefer to use (short) commit IDs, since changing them is infeasible.
  - xen0 6y ago
    I'm assuming the repo, once hosted and the paper is published, is "fixed" and cannot be changed by the authors. But commit ids work just as well.
- mnw21cam 6y ago
  For a paper I recently submitted, the journal demanded a github release of the software.
- jhrmnn 6y ago
  I should have mentioned that I of course use Git. But the need to manually keep track of the calculation~commit pairing is tiresome and error-prone.
qppo 6y ago
That's no different than normal software engineering. We use version control software (VCS, like git) to deal with it. You can include your results in the tracked source. For what it's worth, using results from outdated source code is extremely suspicious. This is a frequent problem in software development where we have tests or benchmarks based on stale code, and it's almost always incorrect. I would not trust your results if they are not created with the most up to date version of your software at all.
chriswarbo 6y ago
I tend to do the following (some or all, depending on the situation): - Use known, plaintext formats like LaTeX, Markdown, CSV, JSON, etc. rather than undocumented binary formats like those of Word, Excel, etc. - Keep sources in git (just a master branch will do) - Write all of the rendering steps into a shell script of Makefile, so it's just one command with no options - I go even further and use Nix, with all dependencies pinned (this is like an extreme form of Make) - Code for generating diagrams, graphs, tables, etc. is kept in git alongside the LaTex/whatever - Generated diagrams/graphs/tables are not included in git; they're generated during rendering, as part of the shell-script/Makefile/Nix-file; the latter only re-generate things if their dependencies have changed - All code is liberally sprinkled with assertions, causing a hard crash if anything looks wrong - If journals/collaborators/etc. want things a certain way (e.g. a zip file containing plain LaTeX, with all diagrams as separate PNGs, or whatever) then the "rendering" should take care of generating that (and make assertions about the result, e.g. that it renders to PDF without error, contains the number of pages we're expecting, that the images have the expected dimensions, etc.) - I push changes from my working copies into a 'repos' directory, which in turn pushes to my Web server and to github (for backups and redundancy) - Pushing changes also triggers a build on the continuous integration server (Laminar) running on my laptop. This makes a fresh copy of the repo and tries to render the document (this prevents depending on uncommitted files, the absolute directory path, etc.) Referencing a particular git commit should be enough to recreate the document (this can also be embedded in the resulting document somewhere, for easy reference). Some care needs to be taken to avoid implicit dependencies, etc. but Nix makes this much easier. Results should also be deterministic; if we need pseudorandom numbers then a fixed seed can be used, or (to prove there's nothing up our sleeves) we can use SHA256 on something that changes on each commit (e.g. the LaTeX source). For computationally-expensive operations (with relatively small outputs) I'll split this across a few git repos: 1) The code for setting up and performing the experiments/generating the data goes in one repo. This is just like any other software project. 2) The results of each experiment/run are kept in a separate git repo. This may be a bad idea for large, binary files; but I've found it works fine for compressed JSON weighing many MBs. Results are always appended to this repo as new files; existing files are never altered, so we don't need to worry about binary diffs. There should be metadata alongside/inside each file which gives the git commit of the experiment repo (i.e. step 1) that was used, alongside other relevant information like machine specs (if it depends on performance), etc. This could be as simple as a file naming scheme. The exact details for this should be written down in this repo, e.g. in a README and/or a simple script to grab the relevant experiment repo, run it, and store the results+metadata in the relevant place. Results should be as "raw" as possible, so that they don't depend on e.g. post-processing details, or choice of analysis, etc. 3) I tend to put the writeup in a separate git repo from the results, so that those results can be referenced by commit + filename, without a load of unrelated churn from the writeup. This repo will follow the same advice as above, e.g. code for turning the "raw" results into graphs, tables, etc. will be kept here and run as part of the rendering process. Fetching the particular commit from the results repo should also be one of the rendering steps (Nix makes this easy, or you could use a git submodule, etc.) I don't know what the best advice is w.r.t. large datasets (GBs or TBs), but I've found the above to be robust for about 5 years so far.

by jhrmnn 6y ago

xen0 6y ago

My first thought: Demand the journals provide hosting for a code repo that is part of your paper. For every numerical result, specify the version (e.g. a git tag) used to generate your result. And if that means scientists need to learn about version control, well... they should if they're writing code.

chriswarbo 6y ago
I agree, except that AFAIK "tags" in git are not fixed, they can be deleted and re-created to point at a different commit. Hence I prefer to use (short) commit IDs, since changing them is infeasible.
- xen0 6y ago
  I'm assuming the repo, once hosted and the paper is published, is "fixed" and cannot be changed by the authors. But commit ids work just as well.
mnw21cam 6y ago
For a paper I recently submitted, the journal demanded a github release of the software.
jhrmnn 6y ago
I should have mentioned that I of course use Git. But the need to manually keep track of the calculation~commit pairing is tiresome and error-prone.

qppo 6y ago

That's no different than normal software engineering. We use version control software (VCS, like git) to deal with it. You can include your results in the tracked source. For what it's worth, using results from outdated source code is extremely suspicious. This is a frequent problem in software development where we have tests or benchmarks based on stale code, and it's almost always incorrect. I would not trust your results if they are not created with the most up to date version of your software at all.

chriswarbo 6y ago

I tend to do the following (some or all, depending on the situation): - Use known, plaintext formats like LaTeX, Markdown, CSV, JSON, etc. rather than undocumented binary formats like those of Word, Excel, etc. - Keep sources in git (just a master branch will do) - Write all of the rendering steps into a shell script of Makefile, so it's just one command with no options - I go even further and use Nix, with all dependencies pinned (this is like an extreme form of Make) - Code for generating diagrams, graphs, tables, etc. is kept in git alongside the LaTex/whatever - Generated diagrams/graphs/tables are not included in git; they're generated during rendering, as part of the shell-script/Makefile/Nix-file; the latter only re-generate things if their dependencies have changed - All code is liberally sprinkled with assertions, causing a hard crash if anything looks wrong - If journals/collaborators/etc. want things a certain way (e.g. a zip file containing plain LaTeX, with all diagrams as separate PNGs, or whatever) then the "rendering" should take care of generating that (and make assertions about the result, e.g. that it renders to PDF without error, contains the number of pages we're expecting, that the images have the expected dimensions, etc.) - I push changes from my working copies into a 'repos' directory, which in turn pushes to my Web server and to github (for backups and redundancy) - Pushing changes also triggers a build on the continuous integration server (Laminar) running on my laptop. This makes a fresh copy of the repo and tries to render the document (this prevents depending on uncommitted files, the absolute directory path, etc.) Referencing a particular git commit should be enough to recreate the document (this can also be embedded in the resulting document somewhere, for easy reference). Some care needs to be taken to avoid implicit dependencies, etc. but Nix makes this much easier. Results should also be deterministic; if we need pseudorandom numbers then a fixed seed can be used, or (to prove there's nothing up our sleeves) we can use SHA256 on something that changes on each commit (e.g. the LaTeX source). For computationally-expensive operations (with relatively small outputs) I'll split this across a few git repos: 1) The code for setting up and performing the experiments/generating the data goes in one repo. This is just like any other software project. 2) The results of each experiment/run are kept in a separate git repo. This may be a bad idea for large, binary files; but I've found it works fine for compressed JSON weighing many MBs. Results are always appended to this repo as new files; existing files are never altered, so we don't need to worry about binary diffs. There should be metadata alongside/inside each file which gives the git commit of the experiment repo (i.e. step 1) that was used, alongside other relevant information like machine specs (if it depends on performance), etc. This could be as simple as a file naming scheme. The exact details for this should be written down in this repo, e.g. in a README and/or a simple script to grab the relevant experiment repo, run it, and store the results+metadata in the relevant place. Results should be as "raw" as possible, so that they don't depend on e.g. post-processing details, or choice of analysis, etc. 3) I tend to put the writeup in a separate git repo from the results, so that those results can be referenced by commit + filename, without a load of unrelated churn from the writeup. This repo will follow the same advice as above, e.g. code for turning the "raw" results into graphs, tables, etc. will be kept here and run as part of the rendering process. Fetching the particular commit from the results repo should also be one of the rendering steps (Nix makes this easy, or you could use a git submodule, etc.) I don't know what the best advice is w.r.t. large datasets (GBs or TBs), but I've found the above to be robust for about 5 years so far.