5 ms·

I am not convinced with the reproducability issues. To me, to reproduce a scientific finding using the exact same versions of all softwares involved is obvious

by anc84 8y ago

I am not convinced with the reproducability issues. To me, to reproduce a scientific finding using the exact same versions of all softwares involved is obvious (unless the authors rule out specific side-effects). So to reproduce paper A from 2015 that used numpy 1.234 in Python 2.654 on a 32-bit Windows system and version abcxyz123 of obscure library X, I would need to recreate those exact conditions. Whatever happened in the future of the softwares is irrelevant. Bugfixes might change results anyways.

eesmith 8y ago
The author wrote this in the comments, and it fits as a reply to your comment: > the main issue is that you cannot suppose that everyone (all program authors and users) know exactly what to do and do it correctly. In practice, the approach you describe almost never works because some information is missing. To make it practical, we'd need easy-to-use tooling for all phases: producing a complete list of versioned dependencies (including C libraries), verifying the completeness of this list, and restoring the environment on a different machine.
- leohutson 8y ago
  So basically a container or VM image. To be honest the python version seems like the smallest issue for reproducibility. I would be more worried about different floating point operations with respect to GPUs for instance.
  - eesmith 8y ago
    The author addressed "a container or VM image" in the followup, at http://blog.khinsen.net/posts/2017/11/22/stability-in-the-scipy-ecosystem-a-summary-of-the-discussion/ http://blog.khinsen.net/posts/2017/11/22/stability-in-the-sc... : > There are three main reasons why this is not a sufficient solution: > 1. Freezing code is fine for archival reproducibility, as I mentioned in my original post. It is not sufficient for living code bases that people work on over decades. ... (His example was his MMTK project which started in the 1.3 days. Twenty years later, it will take a lot of work to make the 2->3 transition.) > 2. The technical solutions proposed all depend on yet more infrastructure whose longevity is uncertain. For how long will a Docker container image produced in 2017 remain usable? For how long will conda and its repositories be supported, and for how long will the binaries in these repositories function on current platforms? > 3. None of today’s code freezing approaches comes with easy-to-use tooling and clear documentation that make it accessible to the average computational scientist. The technologies are today in a “good for early adopters” state. This means we cannot rely on them to preserve today’s research even though they may well take on this role in the future. There are many issues related to reproducibility. I think it's acceptable for different people to focus on different pain points, and that one need not mention all of the possible pain points that others may have when talking about one's own concerns.
    - anc84 8y ago
      1. That's not what reproducibility is about imo. Living code bases are a whole different game. 2. True longevity is a fully self-contained system image. Of course relying on third-parties is not a good idea. 3. Install VirtualBox, doubleclick image, run.
lostcolony 8y ago
Yeah, that'll work when papers stop being accepted/published/paid attention to unless the author's publish a container/image, or at least an environment.txt file listing out all dependencies, and a Python version. I'm working through this now, actually, trying to reproduce some research from last year, and between fixing things that changed between Pytorch 0.3 and 0.4, and cleaning up some of hte underlying data, I've wasted a couple of days (the fixes were easy; figuring out what caused the problems in the first place, not so much).
itronitron 8y ago
alternatively, scientific results should be robust enough to still be observable with marginally different software versions
- plaidfuji 8y ago
  THIS
- maltalex 8y ago
  Not sure if that’s possible in practice. Every analysis would have to be written using several different software stacks... It’s probably doable only if the software is fairly simple or if someone has a lot if extra funding to burn through.
  - hyperbovine 8y ago
    Key word there is marginally. If your finding holds up using NumPy 1.10 but not 1.11... you do not have a finding.
    - hogu 8y ago
      That's probably true for numpy 1.10-1.11, but I"ve seen other libraries make dramatic changes (for example changing the default behavior of a function) http://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.ewma.html http://pandas.pydata.org/pandas-docs/version/0.17.0/generate... pandas changed NaN handling for ewma pre 0.15, which can greatly change the output if your data is sparse
      - hyperbovine 8y ago
        But again, if there are NaN's cropping up in your analysis, and your results depend on what an intermediate tool decides to do with them, you have overlooked something substantial.
BrendanD 8y ago
Imagine if BLAS had to be frozen and run from a container.

by anc84 8y ago

eesmith 8y ago

The author wrote this in the comments, and it fits as a reply to your comment: > the main issue is that you cannot suppose that everyone (all program authors and users) know exactly what to do and do it correctly. In practice, the approach you describe almost never works because some information is missing. To make it practical, we'd need easy-to-use tooling for all phases: producing a complete list of versioned dependencies (including C libraries), verifying the completeness of this list, and restoring the environment on a different machine.

leohutson 8y ago
So basically a container or VM image. To be honest the python version seems like the smallest issue for reproducibility. I would be more worried about different floating point operations with respect to GPUs for instance.
- eesmith 8y ago
  The author addressed "a container or VM image" in the followup, at http://blog.khinsen.net/posts/2017/11/22/stability-in-the-scipy-ecosystem-a-summary-of-the-discussion/ http://blog.khinsen.net/posts/2017/11/22/stability-in-the-sc... : > There are three main reasons why this is not a sufficient solution: > 1. Freezing code is fine for archival reproducibility, as I mentioned in my original post. It is not sufficient for living code bases that people work on over decades. ... (His example was his MMTK project which started in the 1.3 days. Twenty years later, it will take a lot of work to make the 2->3 transition.) > 2. The technical solutions proposed all depend on yet more infrastructure whose longevity is uncertain. For how long will a Docker container image produced in 2017 remain usable? For how long will conda and its repositories be supported, and for how long will the binaries in these repositories function on current platforms? > 3. None of today’s code freezing approaches comes with easy-to-use tooling and clear documentation that make it accessible to the average computational scientist. The technologies are today in a “good for early adopters” state. This means we cannot rely on them to preserve today’s research even though they may well take on this role in the future. There are many issues related to reproducibility. I think it's acceptable for different people to focus on different pain points, and that one need not mention all of the possible pain points that others may have when talking about one's own concerns.
  - anc84 8y ago
    1. That's not what reproducibility is about imo. Living code bases are a whole different game. 2. True longevity is a fully self-contained system image. Of course relying on third-parties is not a good idea. 3. Install VirtualBox, doubleclick image, run.

lostcolony 8y ago

Yeah, that'll work when papers stop being accepted/published/paid attention to unless the author's publish a container/image, or at least an environment.txt file listing out all dependencies, and a Python version. I'm working through this now, actually, trying to reproduce some research from last year, and between fixing things that changed between Pytorch 0.3 and 0.4, and cleaning up some of hte underlying data, I've wasted a couple of days (the fixes were easy; figuring out what caused the problems in the first place, not so much).

itronitron 8y ago

alternatively, scientific results should be robust enough to still be observable with marginally different software versions

plaidfuji 8y ago
THIS
maltalex 8y ago
Not sure if that’s possible in practice. Every analysis would have to be written using several different software stacks... It’s probably doable only if the software is fairly simple or if someone has a lot if extra funding to burn through.
- hyperbovine 8y ago
  Key word there is marginally. If your finding holds up using NumPy 1.10 but not 1.11... you do not have a finding.
  - hogu 8y ago
    That's probably true for numpy 1.10-1.11, but I"ve seen other libraries make dramatic changes (for example changing the default behavior of a function) http://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.ewma.html http://pandas.pydata.org/pandas-docs/version/0.17.0/generate... pandas changed NaN handling for ewma pre 0.15, which can greatly change the output if your data is sparse
    - hyperbovine 8y ago
      But again, if there are NaN's cropping up in your analysis, and your results depend on what an intermediate tool decides to do with them, you have overlooked something substantial.

BrendanD 8y ago

Imagine if BLAS had to be frozen and run from a container.