4 ms·

In a perfect world, studies should not only be required to include the code, any of the following should also be a reason to reject a publication: * The code c

by ufmace 2y ago

In a perfect world, studies should not only be required to include the code, any of the following should also be a reason to reject a publication: * The code can't be run outside the original researcher's undocumented special snowflake system * A fellow scientist who understands the domain can't understand what the code is doing and so is unable to verify that it isn't making any mistakes * It'd be really nice if there were some sort of test cases that could verify that the code in question produced results in line with what the scope of the paper in question considers to be established science in that domain before it was applied to the experimental data - probably not always possible, but it'd be good to at least try Yeah I know this doesn't touch the dozens of other reasons the whole institution is in trouble, and for those reasons usually nobody who has the skill and time to properly review the code actually reviews it carefully. For that matter, most scientists aren't very good at writing code that's easy to understand and will run everywhere. Just one more set of things that ought to go on the list of things that really should be changed.

_yb2s 2y ago
Most scientists unfortunately lack the technical skill to make their code easy to run elsewhere... however I think it is getting better. Someday I would like to see it a requirement that the code has to run anywhere with a single command, e.g. 'docker compose up' or such, and then automatically produce all of the figures and analysis in the paper. There have been some pushes to start journals where the articles themselves can't have any plots or numbers directly, but must automatically generate them from code on the journals servers, e.g. with something like R markdown. It hasn't caught on because the work and technical skill required is too much.
- jltsiren 2y ago
  That's not the code from my point of view. That's the script that processes the output of the code. The actual code often needs nontrivial time and/or money to run, and you may need to spend some effort to match the resources you have with the resources the code needs.
  - _yb2s 2y ago
    The vast majority of papers, even computational ones can run in that way. If there are compute intensive steps, the same process can be done, with a step that includes the previous output, such that it only gets regenerated if the user manually deletes it. I usually do this with Makefiles.

by ufmace 2y ago

_yb2s 2y ago

Most scientists unfortunately lack the technical skill to make their code easy to run elsewhere... however I think it is getting better. Someday I would like to see it a requirement that the code has to run anywhere with a single command, e.g. 'docker compose up' or such, and then automatically produce all of the figures and analysis in the paper. There have been some pushes to start journals where the articles themselves can't have any plots or numbers directly, but must automatically generate them from code on the journals servers, e.g. with something like R markdown. It hasn't caught on because the work and technical skill required is too much.

jltsiren 2y ago
That's not the code from my point of view. That's the script that processes the output of the code. The actual code often needs nontrivial time and/or money to run, and you may need to spend some effort to match the resources you have with the resources the code needs.
- _yb2s 2y ago
  The vast majority of papers, even computational ones can run in that way. If there are compute intensive steps, the same process can be done, with a step that includes the previous output, such that it only gets regenerated if the user manually deletes it. I usually do this with Makefiles.