3 ms·

> They don't show their code in their papers, but the plots/tables. Which is a reproducibility crisis in and of itself, as you mentioned, if not an outright in

• taeric 4y ago
  Strictly, not really. Consider, we rarely (ever?) see the actual needles that go into taking samples. Or the conditions in the polling place where surveys were given/taken. Worse, reproducibility in the "if you click go on this project, you will see it gives the answers I say it does," is of dubious value. Rather, the reproduction is supposed to be "use your models and change assumptions in such a way, you should see these updates." We are at a time where exact reproduction feels possible, but that is typically not as useful as you'd think. A lot of the robustness of old experiments comes from building them all over again.
• TrackerFF 4y ago
  Tbh, you'd need a lot more than just the code to reproduce results today. Last year I spent a couple of weeks getting some academic code to work, due to dependency issues. Rolling back drivers, rolling back libraries, trying specific combinations of drivers and library versions, and what have you. How fast the code ran also depended on the various combinations. The code was not ancient, but written 12 months prior to me trying to build it. (Deep Learning project)
  • adgjlsfhk1 4y ago
    This is one of the places that IMO Julia really shines. It uses a combination of a Project.toml to track dependencies and a Manifest.toml that tracks all the versions of the entire dependency tree which makes it trivial to install all of the package versions.
    • msla 4y ago
      Even that isn't sufficient for archival purposes, which an academic journal should be aiming for.
      • rsfern 4y ago
        What is missing? Is this confined to the software environment, or also covering like data accessibility and workflow documentation? Asking as a scientist interested in reproducible research. Julia has been so much better than python at least in terms of dependency hell, but I still rely on some python packages through PyCall and I feel like that isn’t handled as nicely as Pkg.jl…
        • msla 4y ago
          Just having references to the packages isn't enough, you also need the source code of the packages themselves.
          • ChrisRackauckas 4y ago
            It contains a reference to source code in the package server that is disconnected from the package author so it's retrievable in the event of a left-pad-like event. This is what's queried upon download in order to ensure that no funny business could lead to a security threat down the line either. It's kept on the package servers as an immutable source of truth referenced by the UUID and SHA.
          • adgjlsfhk1 4y ago
            that all works. The package manager tracks source code for every version of every package.