4 ms·

My impression is that there's lots of different niches that have developed, each with their own needs, and trying to unify them is a mess. If you're putting so

by MrJohz 3y ago

My impression is that there's lots of different niches that have developed, each with their own needs, and trying to unify them is a mess. If you're putting something like a web app or something else that will be bundled and distributed as a unit, then you're probably best off with something like Poetry, PDM, or pip-tools - you have a lock file for deterministic dependencies, most of your dependencies will be pure python wheels, and you only really need to test things once. On the other hand, if you're developing a library, you'll need to test against multiple versions of Python, and ideally multiple versions of some of your larger dependencies, to ensure that your library will work for as many users as possible. You'll also need to be able to build and package wheels. Alternatively, you're working in data science, and your main concern is probably making sure you can install the packages you need in the environments you're going to use them - specifically, so that they work with the GPUs and other hardware you have available. And there's still the group of people writing mainly scripts for server maintenance or other tasks, who want to be able to easily install dependencies and keep those dependencies up to date for security reasons, with the minimum number of breaking changes. Right now, there are different tools, packaging systems, etc catering to each of these groups, and so building the One Ring of Python package management is going to involve (a) solving all of these problems, and (b) convincing all these groups of people that your general solution is better than their niche-specific solution. That's certainly not easy, I don't even know if it's all that possible. I do think that working from the ground up (i.e building the individual components like the package metadata file, or the pypackages folder experiment) seems to be working well, in that tools seem to be coalescing around these options and finding the best ways to use them, which is all work that might hopefully feed into new official tooling. But we'll see.

golergka 3y ago
> you're working in data science, and your main concern is probably making sure you can install the packages you need in the environments you're going to use them Honest question from a web developer who sometimes has to work with Python — don't containers solve exactly this?
- MrJohz 3y ago
  Unfortunately no, the problem here is that you're probably going to need a lot of compiled extensions, and some of these extensions are going to be running on your GPU (especially if you're in the ML world, but also more generally if you want to take advantage of e.g. your lab's HPC cluster). PyPI can manage some of this with the wheels system (i.e. OS, architecture, Python ABI), but there's no metadata to indicate, for example, which GPU you have available. So in most cases it's possible to just precompile all the relevant variants and let people download the best one for them, or even in some cases allow people to compile everything for themselves, but there's still situations where those aren't good options. This is why PyTorch is famously more complicated to install via newer packages managers such as Poetry, because it requires something slightly more complicated than the existing Wheel setup, and most package managers aren't designed for that. (Pip isn't designed for that either, but PyTorch has come up with workarounds for pip already.) Containers can't solve this problem because containers are tied to the architecture of the machine they're running on, they can't abstract that away. So even if your code is running in a container, it still needs to know which architecture, OS, resources, etc it has access to.
- buildbot 3y ago
  Not exactly no - https://stackoverflow.com/questions/63960319/does-it-matter-if-the-version-of-cuda-on-docker-is-different-from-the-version-of https://stackoverflow.com/questions/63960319/does-it-matter-... You need to be running a GPU driver on the host that supports the container cuda version. So in theory yes, in practice, weird issue occur sometimes that really suck to debug. For example why do I get NaN loss after spending 8 days on 128 GPUs with this specific set of drivers+cuda container? (Don't hold it that way, use a matching cuda version...) Also a lot of data scientists HATE sys-admin tasks and docker falls squarely into that for many people.
- guappa 3y ago
  The problem is that people doing data science are not developers, so instead of just using whatever is there they are reinventing a terrible version of package management.
wiredfool 3y ago
As I see it, there is a hierarchy of packaging needs, the base levels have been solved over and over with new, better, shiny all you need tools -- while the most tricky, complicated part has been solved over and over separately with each project. * Pure python -- Easy, use one of the declarative ones. * Python + Standalone C -- not too bad, use the build tool. * Python + external (potentially distro supplied) C libraries -- Using setup.py, customized, and different for each project. That last one is where Pillow, the ML space, scipy, and others live, and it's painful. Pillow has a 1000 line setup.py file to find all the optional (and 2 required) dependencies and headers on it's platforms. We've also got code to build the dependencies if necessary for packaging. To port this to some standard, we'd effectively need rpm or dpkg style build infra from PyPa, to work on all the supported platforms.
- MrJohz 3y ago
  I think that's also a valid view of the problem (i.e. the further away from pure Python you get, the more complex and unclear things are). But I also think that's just the view from the "library developers" perspective — if you aren't publishing a library, but using Python for some other purpose, you are going to run into issues even at points that, from your perspective, are already fairly solved. For example, for application developers, even if they just stick to pure Python dependencies, there's still no standard lockfile format that standard Python tools can just emit and ingest. At best, you've got `pip freeze`, but you'll need to use custom tooling to update and maintain that, or switch to pip-compile or another, more full-featured package manager. To me, lockfiles really are table stakes here, but they're not at all easy to get working in Python.
- eesmith 3y ago
  > Python + Standalone C -- not too bad, use the build tool Doesn't that generally end up using setup.py as well? From my understanding, build ends up calling the build-backend, which defaults to setuptools.build_meta:__legacy__, which is setup.py. I know there are other backends, but they seem very specialized to a certain project's needs. I think there's a cmake backend too, but I don't like requiring my customers to install cmake first, and that dependency can't be expressed in pyproject.toml. I had hoped that redo (https://redo.readthedocs.io/en/latest/ https://redo.readthedocs.io/en/latest/) would become popular, as a small, simple, pure-Python Makefile replacement, and that there would be a back-end using it, but neither happened. My specific needs for a backend is to support and configure a code-generation step when building my C extension. The full code generation is >10MB, which handles all 3x24 or so different specialized implementations of the core algorithm. This takes a while to compile, so during development I use a slower, general-purpose implementation.

by MrJohz 3y ago

golergka 3y ago

> you're working in data science, and your main concern is probably making sure you can install the packages you need in the environments you're going to use them Honest question from a web developer who sometimes has to work with Python — don't containers solve exactly this?

MrJohz 3y ago
Unfortunately no, the problem here is that you're probably going to need a lot of compiled extensions, and some of these extensions are going to be running on your GPU (especially if you're in the ML world, but also more generally if you want to take advantage of e.g. your lab's HPC cluster). PyPI can manage some of this with the wheels system (i.e. OS, architecture, Python ABI), but there's no metadata to indicate, for example, which GPU you have available. So in most cases it's possible to just precompile all the relevant variants and let people download the best one for them, or even in some cases allow people to compile everything for themselves, but there's still situations where those aren't good options. This is why PyTorch is famously more complicated to install via newer packages managers such as Poetry, because it requires something slightly more complicated than the existing Wheel setup, and most package managers aren't designed for that. (Pip isn't designed for that either, but PyTorch has come up with workarounds for pip already.) Containers can't solve this problem because containers are tied to the architecture of the machine they're running on, they can't abstract that away. So even if your code is running in a container, it still needs to know which architecture, OS, resources, etc it has access to.
buildbot 3y ago
Not exactly no - https://stackoverflow.com/questions/63960319/does-it-matter-if-the-version-of-cuda-on-docker-is-different-from-the-version-of https://stackoverflow.com/questions/63960319/does-it-matter-... You need to be running a GPU driver on the host that supports the container cuda version. So in theory yes, in practice, weird issue occur sometimes that really suck to debug. For example why do I get NaN loss after spending 8 days on 128 GPUs with this specific set of drivers+cuda container? (Don't hold it that way, use a matching cuda version...) Also a lot of data scientists HATE sys-admin tasks and docker falls squarely into that for many people.
guappa 3y ago
The problem is that people doing data science are not developers, so instead of just using whatever is there they are reinventing a terrible version of package management.

wiredfool 3y ago

As I see it, there is a hierarchy of packaging needs, the base levels have been solved over and over with new, better, shiny all you need tools -- while the most tricky, complicated part has been solved over and over separately with each project. * Pure python -- Easy, use one of the declarative ones. * Python + Standalone C -- not too bad, use the build tool. * Python + external (potentially distro supplied) C libraries -- Using setup.py, customized, and different for each project. That last one is where Pillow, the ML space, scipy, and others live, and it's painful. Pillow has a 1000 line setup.py file to find all the optional (and 2 required) dependencies and headers on it's platforms. We've also got code to build the dependencies if necessary for packaging. To port this to some standard, we'd effectively need rpm or dpkg style build infra from PyPa, to work on all the supported platforms.

MrJohz 3y ago
I think that's also a valid view of the problem (i.e. the further away from pure Python you get, the more complex and unclear things are). But I also think that's just the view from the "library developers" perspective — if you aren't publishing a library, but using Python for some other purpose, you are going to run into issues even at points that, from your perspective, are already fairly solved. For example, for application developers, even if they just stick to pure Python dependencies, there's still no standard lockfile format that standard Python tools can just emit and ingest. At best, you've got `pip freeze`, but you'll need to use custom tooling to update and maintain that, or switch to pip-compile or another, more full-featured package manager. To me, lockfiles really are table stakes here, but they're not at all easy to get working in Python.
eesmith 3y ago
> Python + Standalone C -- not too bad, use the build tool Doesn't that generally end up using setup.py as well? From my understanding, build ends up calling the build-backend, which defaults to setuptools.build_meta:__legacy__, which is setup.py. I know there are other backends, but they seem very specialized to a certain project's needs. I think there's a cmake backend too, but I don't like requiring my customers to install cmake first, and that dependency can't be expressed in pyproject.toml. I had hoped that redo (https://redo.readthedocs.io/en/latest/ https://redo.readthedocs.io/en/latest/) would become popular, as a small, simple, pure-Python Makefile replacement, and that there would be a back-end using it, but neither happened. My specific needs for a backend is to support and configure a code-generation step when building my C extension. The full code generation is >10MB, which handles all 3x24 or so different specialized implementations of the core algorithm. This takes a while to compile, so during development I use a slower, general-purpose implementation.