4 ms·

Packages have always been Python's Achilles heel. Python's philosophy for code is, "there should be one, and preferably only one, obvious way to do it." But f

Packages have always been Python's Achilles heel. Python's philosophy for code is, "there should be one, and preferably only one, obvious way to do it." But for packaging systems, their philosophy is more like Perl: "there's more than one way to do it (and all of them have their own pitfalls)." I don't understand why the Python leadership hasn't shown stronger... leadership... in which tools they recommend.

MrJohz 3y ago
My impression is that there's lots of different niches that have developed, each with their own needs, and trying to unify them is a mess. If you're putting something like a web app or something else that will be bundled and distributed as a unit, then you're probably best off with something like Poetry, PDM, or pip-tools - you have a lock file for deterministic dependencies, most of your dependencies will be pure python wheels, and you only really need to test things once. On the other hand, if you're developing a library, you'll need to test against multiple versions of Python, and ideally multiple versions of some of your larger dependencies, to ensure that your library will work for as many users as possible. You'll also need to be able to build and package wheels. Alternatively, you're working in data science, and your main concern is probably making sure you can install the packages you need in the environments you're going to use them - specifically, so that they work with the GPUs and other hardware you have available. And there's still the group of people writing mainly scripts for server maintenance or other tasks, who want to be able to easily install dependencies and keep those dependencies up to date for security reasons, with the minimum number of breaking changes. Right now, there are different tools, packaging systems, etc catering to each of these groups, and so building the One Ring of Python package management is going to involve (a) solving all of these problems, and (b) convincing all these groups of people that your general solution is better than their niche-specific solution. That's certainly not easy, I don't even know if it's all that possible. I do think that working from the ground up (i.e building the individual components like the package metadata file, or the pypackages folder experiment) seems to be working well, in that tools seem to be coalescing around these options and finding the best ways to use them, which is all work that might hopefully feed into new official tooling. But we'll see.
- golergka 3y ago
  > you're working in data science, and your main concern is probably making sure you can install the packages you need in the environments you're going to use them Honest question from a web developer who sometimes has to work with Python — don't containers solve exactly this?
  - MrJohz 3y ago
    Unfortunately no, the problem here is that you're probably going to need a lot of compiled extensions, and some of these extensions are going to be running on your GPU (especially if you're in the ML world, but also more generally if you want to take advantage of e.g. your lab's HPC cluster). PyPI can manage some of this with the wheels system (i.e. OS, architecture, Python ABI), but there's no metadata to indicate, for example, which GPU you have available. So in most cases it's possible to just precompile all the relevant variants and let people download the best one for them, or even in some cases allow people to compile everything for themselves, but there's still situations where those aren't good options. This is why PyTorch is famously more complicated to install via newer packages managers such as Poetry, because it requires something slightly more complicated than the existing Wheel setup, and most package managers aren't designed for that. (Pip isn't designed for that either, but PyTorch has come up with workarounds for pip already.) Containers can't solve this problem because containers are tied to the architecture of the machine they're running on, they can't abstract that away. So even if your code is running in a container, it still needs to know which architecture, OS, resources, etc it has access to.
  - buildbot 3y ago
    Not exactly no - https://stackoverflow.com/questions/63960319/does-it-matter-if-the-version-of-cuda-on-docker-is-different-from-the-version-of https://stackoverflow.com/questions/63960319/does-it-matter-... You need to be running a GPU driver on the host that supports the container cuda version. So in theory yes, in practice, weird issue occur sometimes that really suck to debug. For example why do I get NaN loss after spending 8 days on 128 GPUs with this specific set of drivers+cuda container? (Don't hold it that way, use a matching cuda version...) Also a lot of data scientists HATE sys-admin tasks and docker falls squarely into that for many people.
tracnar 3y ago
It seems to stem from the Python packaging story being driven by PyPA, rather than the Python foundation, which appears reluctant to pick a packaging solution and ship it with CPython. setuptools was kind of an in-between, it was included in CPython but in many cases you have to update it or add plugins...
eddyg 3y ago
I’m thinking https://astral.sh/ https://astral.sh/ are going to be the ones to “solve” this.
- preciousoo 3y ago
  Unsure if slick promo or not, but that review by tiangolo instantly piqued my interest
- epicureanideal 3y ago
  Astral and Ruff look great, but what is their monetization strategy, and how do their interests intersect with the interests of the developer community in the long term?
- ri0t 3y ago
  Packaging and linting are two very different topics. Just because someone does A very well, doesn't mean they have any clue about B.
- claytonjy 3y ago
  By joining forces with Pyflow, or a different tool, or making yet another one?
gonzo41 3y ago
Python leadership did the python 2->3 adventure. So..... But yeah, I agree. This is something that golang nails. They supply all the options out of the box.
ximm 3y ago
I believe PEP 517 was how the Python leadership wanted to improve the situation. Unfortunately it backfired hard. It broke the existing systems and further fractured the ecosystem.
enthus1ast 3y ago
Packaging was the reason I left python (in favor of Nim) roughly 10 years ago.
stabbles 3y ago
Python's situation is pathetic. Some Python folks wanted to get rid of the builtin module `distutils`. Got deprecated, and now it's finally removed in Python 3.12. That's a breaking change in a minor release, probably because they're too afraid to do a Python 4 release after the 2->3 debacle. Then they publish a migration guide only as part of the 3.12 release, which is too little too late, cause the majority of the projects hasn't moved away from `distutils` yet. There's a lot of unclarity like "you can just install setuptools, it replaces distutils" (setuptools has a startup script that hacks the search paths so you can still run `import distutils` and get their vendored copy of it). Except nobody knows whether to add a conditional dependency on setuptools then. Why on earth do you have to depend on another package to get a vendored copy of distutils? And a fraction of the projects doesn't care and makes it a user problem: just install setuptools yourself as a system / user site-package, then everything works. (Except it doesn't when setuptools is in PYTHONPATH cause the startup script doesn't run, which happens on nix and other prefix-based package managers). At the end of the day, every Python package has their own ugly solution. All that could have been prevented by the Python folks who insisted on deprecating a builtin module. They could have jumped in and help migrate the popular Python packages away from distutils long ago, to set an example. But nope, they really like to keep things messy.
- guappa 3y ago
  You think that's the only module they dropped? They took away cgi… which I was of course using. And crypt. They don't care about breaking things. People would not migrate to python4 so they just break everything in python3 :D
  - eesmith 3y ago
    The cgi module is still in 3.12. It will be removed in 3.13, which is planned for release on 2024-10-01, so not yet dropped. FWIW, the Python developers have wanted to drop the cgi module for over 20 years. It has no maintainer. Depending on your needs, you might consider: 1) versioning and maintaining your own copy of cgi.py; 2) use something like https://pypi.org/project/legacy-cgi/ https://pypi.org/project/legacy-cgi/ (I have not vetted it) which is a forked copy of cgi and cgitb; or 3) use the CGI capabilities of wsgiref (I used it for one CGI project to allow a future transition away from CGI). As for "crypt" - wow! I haven't seen anyone use that since the 1990s! Again, the code is short, and easy to vendor, except for the _crypt extension module. For that, you might be able to use ctypes, like this: >>> from ctypes.util import find_library >>> find_library("c") '/usr/lib/libc.dylib' >>> from ctypes import cdll >>> libc = cdll.LoadLibrary(find_library("c")) >>> import ctypes >>> libc.crypt.argtypes = [ctypes.c_char_p, ctypes.c_char_p] >>> libc.crypt.restype = ctypes.c_char_p >>> libc.crypt(b"toomanysecrets", b"az") b'azi4LBG1VJohQ' Here is what Python's own _crypt does: >>> import _crypt >>> _crypt.crypt("toomanysecrets", "az") 'azi4LBG1VJohQ'
nor-and-or-not 3y ago
The funny thing in context to this is that Perl has rock solid package management and distribution since like forever (CPAN).
- ok123456 3y ago
  If the code hasn't had any updates in 15 years, it's easy to be rock solid.

MrJohz 3y ago

My impression is that there's lots of different niches that have developed, each with their own needs, and trying to unify them is a mess. If you're putting something like a web app or something else that will be bundled and distributed as a unit, then you're probably best off with something like Poetry, PDM, or pip-tools - you have a lock file for deterministic dependencies, most of your dependencies will be pure python wheels, and you only really need to test things once. On the other hand, if you're developing a library, you'll need to test against multiple versions of Python, and ideally multiple versions of some of your larger dependencies, to ensure that your library will work for as many users as possible. You'll also need to be able to build and package wheels. Alternatively, you're working in data science, and your main concern is probably making sure you can install the packages you need in the environments you're going to use them - specifically, so that they work with the GPUs and other hardware you have available. And there's still the group of people writing mainly scripts for server maintenance or other tasks, who want to be able to easily install dependencies and keep those dependencies up to date for security reasons, with the minimum number of breaking changes. Right now, there are different tools, packaging systems, etc catering to each of these groups, and so building the One Ring of Python package management is going to involve (a) solving all of these problems, and (b) convincing all these groups of people that your general solution is better than their niche-specific solution. That's certainly not easy, I don't even know if it's all that possible. I do think that working from the ground up (i.e building the individual components like the package metadata file, or the pypackages folder experiment) seems to be working well, in that tools seem to be coalescing around these options and finding the best ways to use them, which is all work that might hopefully feed into new official tooling. But we'll see.

golergka 3y ago
> you're working in data science, and your main concern is probably making sure you can install the packages you need in the environments you're going to use them Honest question from a web developer who sometimes has to work with Python — don't containers solve exactly this?
- MrJohz 3y ago
  Unfortunately no, the problem here is that you're probably going to need a lot of compiled extensions, and some of these extensions are going to be running on your GPU (especially if you're in the ML world, but also more generally if you want to take advantage of e.g. your lab's HPC cluster). PyPI can manage some of this with the wheels system (i.e. OS, architecture, Python ABI), but there's no metadata to indicate, for example, which GPU you have available. So in most cases it's possible to just precompile all the relevant variants and let people download the best one for them, or even in some cases allow people to compile everything for themselves, but there's still situations where those aren't good options. This is why PyTorch is famously more complicated to install via newer packages managers such as Poetry, because it requires something slightly more complicated than the existing Wheel setup, and most package managers aren't designed for that. (Pip isn't designed for that either, but PyTorch has come up with workarounds for pip already.) Containers can't solve this problem because containers are tied to the architecture of the machine they're running on, they can't abstract that away. So even if your code is running in a container, it still needs to know which architecture, OS, resources, etc it has access to.
- buildbot 3y ago
  Not exactly no - https://stackoverflow.com/questions/63960319/does-it-matter-if-the-version-of-cuda-on-docker-is-different-from-the-version-of https://stackoverflow.com/questions/63960319/does-it-matter-... You need to be running a GPU driver on the host that supports the container cuda version. So in theory yes, in practice, weird issue occur sometimes that really suck to debug. For example why do I get NaN loss after spending 8 days on 128 GPUs with this specific set of drivers+cuda container? (Don't hold it that way, use a matching cuda version...) Also a lot of data scientists HATE sys-admin tasks and docker falls squarely into that for many people.

tracnar 3y ago

It seems to stem from the Python packaging story being driven by PyPA, rather than the Python foundation, which appears reluctant to pick a packaging solution and ship it with CPython. setuptools was kind of an in-between, it was included in CPython but in many cases you have to update it or add plugins...

eddyg 3y ago

I’m thinking https://astral.sh/ https://astral.sh/ are going to be the ones to “solve” this.

preciousoo 3y ago
Unsure if slick promo or not, but that review by tiangolo instantly piqued my interest
epicureanideal 3y ago
Astral and Ruff look great, but what is their monetization strategy, and how do their interests intersect with the interests of the developer community in the long term?
ri0t 3y ago
Packaging and linting are two very different topics. Just because someone does A very well, doesn't mean they have any clue about B.
claytonjy 3y ago
By joining forces with Pyflow, or a different tool, or making yet another one?

gonzo41 3y ago

Python leadership did the python 2->3 adventure. So..... But yeah, I agree. This is something that golang nails. They supply all the options out of the box.

ximm 3y ago

I believe PEP 517 was how the Python leadership wanted to improve the situation. Unfortunately it backfired hard. It broke the existing systems and further fractured the ecosystem.

enthus1ast 3y ago

Packaging was the reason I left python (in favor of Nim) roughly 10 years ago.

stabbles 3y ago

Python's situation is pathetic. Some Python folks wanted to get rid of the builtin module `distutils`. Got deprecated, and now it's finally removed in Python 3.12. That's a breaking change in a minor release, probably because they're too afraid to do a Python 4 release after the 2->3 debacle. Then they publish a migration guide only as part of the 3.12 release, which is too little too late, cause the majority of the projects hasn't moved away from `distutils` yet. There's a lot of unclarity like "you can just install setuptools, it replaces distutils" (setuptools has a startup script that hacks the search paths so you can still run `import distutils` and get their vendored copy of it). Except nobody knows whether to add a conditional dependency on setuptools then. Why on earth do you have to depend on another package to get a vendored copy of distutils? And a fraction of the projects doesn't care and makes it a user problem: just install setuptools yourself as a system / user site-package, then everything works. (Except it doesn't when setuptools is in PYTHONPATH cause the startup script doesn't run, which happens on nix and other prefix-based package managers). At the end of the day, every Python package has their own ugly solution. All that could have been prevented by the Python folks who insisted on deprecating a builtin module. They could have jumped in and help migrate the popular Python packages away from distutils long ago, to set an example. But nope, they really like to keep things messy.

guappa 3y ago
You think that's the only module they dropped? They took away cgi… which I was of course using. And crypt. They don't care about breaking things. People would not migrate to python4 so they just break everything in python3 :D
- eesmith 3y ago
  The cgi module is still in 3.12. It will be removed in 3.13, which is planned for release on 2024-10-01, so not yet dropped. FWIW, the Python developers have wanted to drop the cgi module for over 20 years. It has no maintainer. Depending on your needs, you might consider: 1) versioning and maintaining your own copy of cgi.py; 2) use something like https://pypi.org/project/legacy-cgi/ https://pypi.org/project/legacy-cgi/ (I have not vetted it) which is a forked copy of cgi and cgitb; or 3) use the CGI capabilities of wsgiref (I used it for one CGI project to allow a future transition away from CGI). As for "crypt" - wow! I haven't seen anyone use that since the 1990s! Again, the code is short, and easy to vendor, except for the _crypt extension module. For that, you might be able to use ctypes, like this: >>> from ctypes.util import find_library >>> find_library("c") '/usr/lib/libc.dylib' >>> from ctypes import cdll >>> libc = cdll.LoadLibrary(find_library("c")) >>> import ctypes >>> libc.crypt.argtypes = [ctypes.c_char_p, ctypes.c_char_p] >>> libc.crypt.restype = ctypes.c_char_p >>> libc.crypt(b"toomanysecrets", b"az") b'azi4LBG1VJohQ' Here is what Python's own _crypt does: >>> import _crypt >>> _crypt.crypt("toomanysecrets", "az") 'azi4LBG1VJohQ'

nor-and-or-not 3y ago

The funny thing in context to this is that Perl has rock solid package management and distribution since like forever (CPAN).

ok123456 3y ago
If the code hasn't had any updates in 15 years, it's easy to be rock solid.