3 ms·

That's why people move to crappy conda / docker. To keep those Python 2 environments running. There no new people to rewrite tons of software.

leephillips 6y ago
There's another reason for using containers or pretested distributions. I just wrote an article for LWN about SciPy¹. To get what was supposed to be a clean install of the newest release, I created a virtual environment and used Pip to install the SciPy components. It was broken out of the box. IPython did not work because a module that it depends on, that does tab completion, had been recently upgraded and was incompatible with it. I mean it crashed frequently, not just that it didn’t do tab completion. To get a working system, I had to install an older version of the library. So people seek solutions that avoid the dependency hell that comes from maintainers releasing things at will, with nobody testing the combination of things that they nevertheless market as “SciPy”, which is supposed to be a bunch of stuff that works together. And whenever I install anything from the Python world, I am amazed when it actually works without the need to spend hours with Google or staring at sources trying to resolve conflicts. One of the many nice things about Julia is that their package system works. So yes, once you manage to put together a working installation, you naturally want to encase that precious thing in the bubble of some kind of container, to protect it from an upgrade of some piece that will break everything. If you work with a lot of Python things, you may have a dozen copies of the same libraries, and a handful of Python binaries, taking up space on your drive. The concept of dynamic linking was supposed to save us from this. But the haphazard habits endemic to the Python community seem to have left us with the worst of all possible worlds. [1] https://lwn.net/SubscriberLink/842964/f7e2e5006c5efe36/ https://lwn.net/SubscriberLink/842964/f7e2e5006c5efe36/
- kortex 6y ago
  Docker is the best thing to ever happen to scientific computing yet there is so much griping from the old guard. Trying to make a hermetic, CI/CD, within-machine-epsilon reproducible computer vision pipeline opened my eyes to just how un-reproducible your average machine learning project really is. Especially those coming from academia. "Oh the PhD pinned to a nightly version of some dep and the AUC score plummets when you try to upgrade and customer wants this latest model in production on airgapped machines last week? Aight, stick their Conda env in a container, call it with `docker run`, and sort it out later" Horrifying? Yes. Solving the problem on short notice? Also yes. Plenty of more pedestrian uses but that was an actual situation where docker's reproducibility saved my bacon.
  - leephillips 6y ago
    Yes. When your calculation uses dozens of files and a change in any of them might change the results, I don’t see how there is any alternative to some form of containerization. Even if you could generate a statically linked binary, if it’s science you want to distribute the sources for everything as well. I notice that most people seem to reach for Docker automatically, though. There are other technologies, such as systemd containers, that might be better in some situations.

leephillips 6y ago

There's another reason for using containers or pretested distributions. I just wrote an article for LWN about SciPy¹. To get what was supposed to be a clean install of the newest release, I created a virtual environment and used Pip to install the SciPy components. It was broken out of the box. IPython did not work because a module that it depends on, that does tab completion, had been recently upgraded and was incompatible with it. I mean it crashed frequently, not just that it didn’t do tab completion. To get a working system, I had to install an older version of the library. So people seek solutions that avoid the dependency hell that comes from maintainers releasing things at will, with nobody testing the combination of things that they nevertheless market as “SciPy”, which is supposed to be a bunch of stuff that works together. And whenever I install anything from the Python world, I am amazed when it actually works without the need to spend hours with Google or staring at sources trying to resolve conflicts. One of the many nice things about Julia is that their package system works. So yes, once you manage to put together a working installation, you naturally want to encase that precious thing in the bubble of some kind of container, to protect it from an upgrade of some piece that will break everything. If you work with a lot of Python things, you may have a dozen copies of the same libraries, and a handful of Python binaries, taking up space on your drive. The concept of dynamic linking was supposed to save us from this. But the haphazard habits endemic to the Python community seem to have left us with the worst of all possible worlds. [1] https://lwn.net/SubscriberLink/842964/f7e2e5006c5efe36/ https://lwn.net/SubscriberLink/842964/f7e2e5006c5efe36/

kortex 6y ago
Docker is the best thing to ever happen to scientific computing yet there is so much griping from the old guard. Trying to make a hermetic, CI/CD, within-machine-epsilon reproducible computer vision pipeline opened my eyes to just how un-reproducible your average machine learning project really is. Especially those coming from academia. "Oh the PhD pinned to a nightly version of some dep and the AUC score plummets when you try to upgrade and customer wants this latest model in production on airgapped machines last week? Aight, stick their Conda env in a container, call it with `docker run`, and sort it out later" Horrifying? Yes. Solving the problem on short notice? Also yes. Plenty of more pedestrian uses but that was an actual situation where docker's reproducibility saved my bacon.
- leephillips 6y ago
  Yes. When your calculation uses dozens of files and a change in any of them might change the results, I don’t see how there is any alternative to some form of containerization. Even if you could generate a statically linked binary, if it’s science you want to distribute the sources for everything as well. I notice that most people seem to reach for Docker automatically, though. There are other technologies, such as systemd containers, that might be better in some situations.