3 ms·

not in my neck of science python (biomed/neuroscience; computer vision). We've been substantially more aggressive about moving to 3 than other python neighborh

not in my neck of science python (biomed/neuroscience; computer vision). We've been substantially more aggressive about moving to 3 than other python neighborhoods! Look at the top projects at https://python3statement.org/ https://python3statement.org/, for example. Hell, numpy 1.20.x is dropping support for anything below 3.7! I started doing science python in 2015 on 3.5.x, never had to touch 2.x at all.

GNU_James 6y ago
That's why people move to crappy conda / docker. To keep those Python 2 environments running. There no new people to rewrite tons of software.
- leephillips 6y ago
  There's another reason for using containers or pretested distributions. I just wrote an article for LWN about SciPy¹. To get what was supposed to be a clean install of the newest release, I created a virtual environment and used Pip to install the SciPy components. It was broken out of the box. IPython did not work because a module that it depends on, that does tab completion, had been recently upgraded and was incompatible with it. I mean it crashed frequently, not just that it didn’t do tab completion. To get a working system, I had to install an older version of the library. So people seek solutions that avoid the dependency hell that comes from maintainers releasing things at will, with nobody testing the combination of things that they nevertheless market as “SciPy”, which is supposed to be a bunch of stuff that works together. And whenever I install anything from the Python world, I am amazed when it actually works without the need to spend hours with Google or staring at sources trying to resolve conflicts. One of the many nice things about Julia is that their package system works. So yes, once you manage to put together a working installation, you naturally want to encase that precious thing in the bubble of some kind of container, to protect it from an upgrade of some piece that will break everything. If you work with a lot of Python things, you may have a dozen copies of the same libraries, and a handful of Python binaries, taking up space on your drive. The concept of dynamic linking was supposed to save us from this. But the haphazard habits endemic to the Python community seem to have left us with the worst of all possible worlds. [1] https://lwn.net/SubscriberLink/842964/f7e2e5006c5efe36/ https://lwn.net/SubscriberLink/842964/f7e2e5006c5efe36/
  - kortex 6y ago
    Docker is the best thing to ever happen to scientific computing yet there is so much griping from the old guard. Trying to make a hermetic, CI/CD, within-machine-epsilon reproducible computer vision pipeline opened my eyes to just how un-reproducible your average machine learning project really is. Especially those coming from academia. "Oh the PhD pinned to a nightly version of some dep and the AUC score plummets when you try to upgrade and customer wants this latest model in production on airgapped machines last week? Aight, stick their Conda env in a container, call it with `docker run`, and sort it out later" Horrifying? Yes. Solving the problem on short notice? Also yes. Plenty of more pedestrian uses but that was an actual situation where docker's reproducibility saved my bacon.
    - leephillips 6y ago
      Yes. When your calculation uses dozens of files and a change in any of them might change the results, I don’t see how there is any alternative to some form of containerization. Even if you could generate a statically linked binary, if it’s science you want to distribute the sources for everything as well. I notice that most people seem to reach for Docker automatically, though. There are other technologies, such as systemd containers, that might be better in some situations.
gh02t 6y ago
Same in my neck of the woods (engineering R&D). Python is very popular and everybody I know is using Python 3 unless they are supporting an outdated codebase, where they are usually in process of migrating.
willis936 6y ago
I work on a physics project. The data acquisition system was made on a tiny budget and most of the code is python 2.5.2. It hasn’t been touched in 12 years. We’re stuck with Windows 7 running on some machines because there are no Python 2.5 libraries that support NI-DAQmx and NI does not have any Traditional NI DAQ releases in windows 10. Obviously good solutions involve porting to linux and airgapping the DAQ, but the simplest solution I like is updating the codebase. I’m one engineer with loads of other responsibilities. Python is popular with scientists. I don’t begrudge the scientists. I begrudge the awful decision by the python developers to disregard backwards compatibility.

GNU_James 6y ago

That's why people move to crappy conda / docker. To keep those Python 2 environments running. There no new people to rewrite tons of software.

leephillips 6y ago
There's another reason for using containers or pretested distributions. I just wrote an article for LWN about SciPy¹. To get what was supposed to be a clean install of the newest release, I created a virtual environment and used Pip to install the SciPy components. It was broken out of the box. IPython did not work because a module that it depends on, that does tab completion, had been recently upgraded and was incompatible with it. I mean it crashed frequently, not just that it didn’t do tab completion. To get a working system, I had to install an older version of the library. So people seek solutions that avoid the dependency hell that comes from maintainers releasing things at will, with nobody testing the combination of things that they nevertheless market as “SciPy”, which is supposed to be a bunch of stuff that works together. And whenever I install anything from the Python world, I am amazed when it actually works without the need to spend hours with Google or staring at sources trying to resolve conflicts. One of the many nice things about Julia is that their package system works. So yes, once you manage to put together a working installation, you naturally want to encase that precious thing in the bubble of some kind of container, to protect it from an upgrade of some piece that will break everything. If you work with a lot of Python things, you may have a dozen copies of the same libraries, and a handful of Python binaries, taking up space on your drive. The concept of dynamic linking was supposed to save us from this. But the haphazard habits endemic to the Python community seem to have left us with the worst of all possible worlds. [1] https://lwn.net/SubscriberLink/842964/f7e2e5006c5efe36/ https://lwn.net/SubscriberLink/842964/f7e2e5006c5efe36/
- kortex 6y ago
  Docker is the best thing to ever happen to scientific computing yet there is so much griping from the old guard. Trying to make a hermetic, CI/CD, within-machine-epsilon reproducible computer vision pipeline opened my eyes to just how un-reproducible your average machine learning project really is. Especially those coming from academia. "Oh the PhD pinned to a nightly version of some dep and the AUC score plummets when you try to upgrade and customer wants this latest model in production on airgapped machines last week? Aight, stick their Conda env in a container, call it with `docker run`, and sort it out later" Horrifying? Yes. Solving the problem on short notice? Also yes. Plenty of more pedestrian uses but that was an actual situation where docker's reproducibility saved my bacon.
  - leephillips 6y ago
    Yes. When your calculation uses dozens of files and a change in any of them might change the results, I don’t see how there is any alternative to some form of containerization. Even if you could generate a statically linked binary, if it’s science you want to distribute the sources for everything as well. I notice that most people seem to reach for Docker automatically, though. There are other technologies, such as systemd containers, that might be better in some situations.

gh02t 6y ago

Same in my neck of the woods (engineering R&D). Python is very popular and everybody I know is using Python 3 unless they are supporting an outdated codebase, where they are usually in process of migrating.

willis936 6y ago

I work on a physics project. The data acquisition system was made on a tiny budget and most of the code is python 2.5.2. It hasn’t been touched in 12 years. We’re stuck with Windows 7 running on some machines because there are no Python 2.5 libraries that support NI-DAQmx and NI does not have any Traditional NI DAQ releases in windows 10. Obviously good solutions involve porting to linux and airgapping the DAQ, but the simplest solution I like is updating the codebase. I’m one engineer with loads of other responsibilities. Python is popular with scientists. I don’t begrudge the scientists. I begrudge the awful decision by the python developers to disregard backwards compatibility.