6 ms·

Python wheel filenames have no canonical form

60 points by woodruffw 2y ago

rvnx 2y ago
[flagged]
- woodruffw 2y ago
  This is an incorrect summary: if you read beyond that, name canonicalization is not a significant hurdle. The larger hurdle is the fact that wheel filenames support compressed tags, which themselves lack canonicalization.
  - deleted 2y ago
    [deleted]
g15jv2dp 2y ago
Why is Python packaging such a mess? That's a serious question. Has someone written up some recap or history piece on how it came to be this way? Any pointers?
- amlozano 2y ago
  Python was first released in the early 90s. If you compare Python packaging to other languages of its time, its not so crazy. For comparison, C++ was released in '85.
  - g15jv2dp 2y ago
    They weren't afraid to break pretty much everything with the release of Python 3 in 2008. That's my baseline.
    - pphysch 2y ago
      And yet Python3 is more popular than ever /shrug
- rvnx 2y ago
  500'000 million packages, times n versions, all of that evolving over 30 years.
- woodruffw 2y ago
  The Python packaging ecosystem is 25+ years old, and was largely informally specified until ~12 years ago. Most of the complexity in Python packaging can be traced back to (1) a large degree of pre-existing permissive behavior that needed to be encoded in early standards to get standardization off the ground, and (2) the fact that Python packaging is still largely federated, with multiple independent groups performing tool development (and interpreting the standards). (2) has historically also been a strength of Python packaging, and has been a big source of modernization efforts over the years that (IMO) have made packaging much better: things like pyproject.toml, build backends, etc. (I think there have been a few recap talks at PyCon over the years on packaging's history. There are definitely people way more qualified than me to provide a more detailed summary :-)).
  - rvnx 2y ago
    One example, Semantic Versioning and odd/even versioning, where 3.1 is considered unstable, versus 3.2 is considered stable, versus 3.3 unstable, etc. This was standardized only in 2010, so we can't blame Python developers for not foreseeing that in 2000.
    - woodruffw 2y ago
      Right -- one of Python (packaging's) small ironies is that people sometimes forget how old the Python ecosystem is, in part because the ecosystem and language have managed to keep up with the times :-) (If we use similarly-aged interpreted languages as a benchmark, I would say that Python packaging has modernized admirably in some aspects: compare CPAN, for example.)
      - rvnx 2y ago
        Oh yes, can't agree more, compared to Perl it has completely modernized and stayed relatively well organized :) It's a very pleasant platform.
      - g15jv2dp 2y ago
        What's wrong with CPAN?
- zackproser 2y ago
  https://www.pinecone.io/blog/pain-poetry-python/ https://www.pinecone.io/blog/pain-poetry-python/ This was the write up I did last year
  - kingkongjaffa 2y ago
    Hey this is a nice write up and I'm glad to be able to follow along, I've been writing Python since around 2012. Poetry is a step in the right direction, but I would love to see it baked into the language itself somehow.
    - maleldil 2y ago
      I don't think we'll ever see proper packaging integrated into the language. Our best hope is for the ecosystem to coalesce around a particular tool, or at least have standards that all the major players follow. The latter is kind of happening slowly over time.
    - doctorpangloss 2y ago
      Poetry is adopted by less than 0.5% of Python repos on GitHub. Because you can’t install PyTorch with it in a way that works for more than 1 platform, and sometimes in general, its adoption among new and sophisticated projects is trending towards zero. I have no beef with Poetry but having taken huge brain damage dealing with Python packaging as a USP for end users, the maintainers have decided to not participate in the most exciting things in the ecosystem to the detriment of the otherwise excellent and thoughtfully designed project. Ultimately people who are offering a packaging alternative must deliver a “universal install wizard” solution that includes Windows, which interests few people to work on.
- 0xbadcafebee 2y ago
  Because the core team never took it seriously. You can let packaging solutions grow organically if your core team doesn't want to do it, but then you need to find those who start a project, help guide them, point people toward them, help it grow and improve. You can't just throw your hands up and hope somebody else solves it. Perl has the best packaging community I've seen. Ironically I don't think they even have binary packages, but they do have a system that tests both core Perl and most of the CPAN archive on like 20+ platforms. The packaging fits the community, it's robust, and sets conventions that the rest of the community follows that improves the entire ecosystem. It has definitely evolved over time. But I don't know of anyone who has ever said "Man, Perl packaging is a huge pain in the butt, I wish someone could solve this". If there's a problem, usually there is just one solution that everyone adopts, builds on and improves (which is ironic considering Perl's motto) Whereas with Python, if there's a problem, someone tries to solve it, but it's not perfect, so somebody makes a second solution, then a third, fourth, fifth, etc. Python never established the convention of reusing and improving existing things. Python has a forking/independent culture. So there will always be 20 different ways to do something and none of them all that great (which is ironic considering Python's principles)
- deleted 2y ago
  [deleted]
- formerly_proven 2y ago
  Most other packaging systems don't even attempt to solve the problems that e.g. wheels solve.
- mjw1007 2y ago
  The early days of setuptools/easy_install (around 2004) were interesting. The way I remember it, the main developer had a clear "don't ask for permission, just build what you think is best and put it out there" approach. That avoids the problems you can get where everything gets discussed to death and nothing actually gets written, but the downside was that the project spent a long time in an uneasy situation where it didn't have complete official buy-in but was popular enough that it would have been hard to push an alternative. There were a couple of questionable early design decisions that didn't help.
wruza 2y ago
There’s also no package caching across venvs(?). Every time I have to install something AI it’s the same 2GB torch + few hundred megabytes libs + cuda update of downloads again and again. On top of that, python projects love to start downloading large resources without asking, and if you ctrl-c it, the setup process is usually non-restartable. You’re just left with a broken folder that doesn’t run anymore. Feels like everything about python’s “install” is an afterthought at best.
- kzrdude 2y ago
  uv by astral is cleaning up and improving a lot in this area right now
  - nubinetwork 2y ago
    Pip, conda, uv... how many package managers do we need?
    - IanCal 2y ago
      Just one, but one that's decent. We'll keep inventing new ones until then. Maybe beyond.
      - cdchn 2y ago
        I don't understand the obsession with reimplementing Python utilities or ecosystem elements in Rust.
    - kzrdude 2y ago
      Uv's adoption rate says that there was a huge demand for it
    - maleldil 2y ago
      uv is essentially an improved version of pip, virtualenv and pip-tools, but from the outside it's pretty much the same thing. The CLI imitates most of pip's options too. It's not like conda, which is a completely different ecosystem, with dodgy integration with pip.
senkora 2y ago
> To allow for compact filenames of bdists that work with more than one compatibility tag triple, each tag in a filename can instead be a ‘.’-separated, sorted, set of tags. Is his second point about compressed tag sets right? I would interpret "sorted, set" to disallow unsorted and duplicate tags.
- woodruffw 2y ago
  Two aspects: 1. Sorted, set is a contraction in terms: `packaging` and other tooling have interpreted it to mean just "set," which implies unsorted. 2. The quoted language there isn't from the wheel filename definition itself (and isn't linked to it), so it's unclear whether it's intended to be normative. Similarly, "can" is weaker than "should."
amelius 2y ago
Also, when installing a wheel you can get a brief error message "Not a supported wheel on this platform" without an explanation of why this wheel is not compatible with your system, which can be infuriating.
- zbentley 2y ago
  Yeah, that's incredibly frustrating. There's an open issue about it which you can express support for, and some marginally-hacky code that you can paste into a "debug_wheel.py"-ish script to get unblocked in the short term, here: https://github.com/pypa/pip/issues/10793 https://github.com/pypa/pip/issues/10793
ethbr1 2y ago
I assumed by now that "Python _____ has no canonical form" was a feature, not a bug, to the Python community. Let them eat cake!
- deleted 2y ago
  [deleted]
- deleted 2y ago
  [deleted]
agumonkey 2y ago
Not helped by every boundaries you'll have to cross to deploy a python lib or app. Pypa will not necessarily agree with pypi and or docker image versionning. It's amazing how fast naming hell travels.
nerdponx 2y ago
Note to readers who enjoy complaining about Python: the title is more bad-sounding than the actual content. It's analogous to how IDNs and IP addresses have no single canonical form, but 2 or more equivalent forms that can be parsed. To properly compare two IP addresses, you must parse them and compare the results. The same thing is happening here. It's an interesting problem and maybe an oversight in the design of the wheel name format, but it's not really an instance of "Python bad" the way you might assume based on the title.
- woodruffw 2y ago
  Right -- I'd consider it strictly a curiosity, and not even a flaw.
zbentley 2y ago
In addition to standardization headaches, the wheel issue discussed is also hindered by using file names as an at least somewhat complex medatata structure's transport. Sure, you can do that, but it's a pain in the ass even if you do a good job. A file name is just one string, after all. This is a somewhat hot take, but I wish there were more widespread use of package repository tools that don't use the dpkg model of "the registry's metadata is baked into a simple file hierarchy". The simplicity and accessibility (I can just browse packages with HTTP/FTP) of that approach is appealing, but it breaks down a lot in cases like this. I don't think it's too opaque or overcomplicated to wish for repository servers that have a pair of access APIs: one to ask the server about metadata/packages matching a set of constraints, and another to download blobs/tarballs/whatever. Docker takes that approach, and it seems to work well. I just wish that instances of this pattern weren't per-language/platform and we had something a little more standardized.
VagabundoP 2y ago
I've been testing out and migrating my hobby projects to Rye/uv and while they are still buggy in some cases i've been pleased with the results. There's still work to do but Rye works well with the standards and conventions as they currently exist.

60 points by woodruffw 2y ago

rvnx 2y ago

[flagged]

woodruffw 2y ago
This is an incorrect summary: if you read beyond that, name canonicalization is not a significant hurdle. The larger hurdle is the fact that wheel filenames support compressed tags, which themselves lack canonicalization.
- deleted 2y ago
  [deleted]

g15jv2dp 2y ago

Why is Python packaging such a mess? That's a serious question. Has someone written up some recap or history piece on how it came to be this way? Any pointers?

amlozano 2y ago
Python was first released in the early 90s. If you compare Python packaging to other languages of its time, its not so crazy. For comparison, C++ was released in '85.
- g15jv2dp 2y ago
  They weren't afraid to break pretty much everything with the release of Python 3 in 2008. That's my baseline.
  - pphysch 2y ago
    And yet Python3 is more popular than ever /shrug
rvnx 2y ago
500'000 million packages, times n versions, all of that evolving over 30 years.
woodruffw 2y ago
The Python packaging ecosystem is 25+ years old, and was largely informally specified until ~12 years ago. Most of the complexity in Python packaging can be traced back to (1) a large degree of pre-existing permissive behavior that needed to be encoded in early standards to get standardization off the ground, and (2) the fact that Python packaging is still largely federated, with multiple independent groups performing tool development (and interpreting the standards). (2) has historically also been a strength of Python packaging, and has been a big source of modernization efforts over the years that (IMO) have made packaging much better: things like pyproject.toml, build backends, etc. (I think there have been a few recap talks at PyCon over the years on packaging's history. There are definitely people way more qualified than me to provide a more detailed summary :-)).
- rvnx 2y ago
  One example, Semantic Versioning and odd/even versioning, where 3.1 is considered unstable, versus 3.2 is considered stable, versus 3.3 unstable, etc. This was standardized only in 2010, so we can't blame Python developers for not foreseeing that in 2000.
  - woodruffw 2y ago
    Right -- one of Python (packaging's) small ironies is that people sometimes forget how old the Python ecosystem is, in part because the ecosystem and language have managed to keep up with the times :-) (If we use similarly-aged interpreted languages as a benchmark, I would say that Python packaging has modernized admirably in some aspects: compare CPAN, for example.)
    - rvnx 2y ago
      Oh yes, can't agree more, compared to Perl it has completely modernized and stayed relatively well organized :) It's a very pleasant platform.
    - g15jv2dp 2y ago
      What's wrong with CPAN?
zackproser 2y ago
https://www.pinecone.io/blog/pain-poetry-python/ https://www.pinecone.io/blog/pain-poetry-python/ This was the write up I did last year
- kingkongjaffa 2y ago
  Hey this is a nice write up and I'm glad to be able to follow along, I've been writing Python since around 2012. Poetry is a step in the right direction, but I would love to see it baked into the language itself somehow.
  - maleldil 2y ago
    I don't think we'll ever see proper packaging integrated into the language. Our best hope is for the ecosystem to coalesce around a particular tool, or at least have standards that all the major players follow. The latter is kind of happening slowly over time.
  - doctorpangloss 2y ago
    Poetry is adopted by less than 0.5% of Python repos on GitHub. Because you can’t install PyTorch with it in a way that works for more than 1 platform, and sometimes in general, its adoption among new and sophisticated projects is trending towards zero. I have no beef with Poetry but having taken huge brain damage dealing with Python packaging as a USP for end users, the maintainers have decided to not participate in the most exciting things in the ecosystem to the detriment of the otherwise excellent and thoughtfully designed project. Ultimately people who are offering a packaging alternative must deliver a “universal install wizard” solution that includes Windows, which interests few people to work on.
0xbadcafebee 2y ago
Because the core team never took it seriously. You can let packaging solutions grow organically if your core team doesn't want to do it, but then you need to find those who start a project, help guide them, point people toward them, help it grow and improve. You can't just throw your hands up and hope somebody else solves it. Perl has the best packaging community I've seen. Ironically I don't think they even have binary packages, but they do have a system that tests both core Perl and most of the CPAN archive on like 20+ platforms. The packaging fits the community, it's robust, and sets conventions that the rest of the community follows that improves the entire ecosystem. It has definitely evolved over time. But I don't know of anyone who has ever said "Man, Perl packaging is a huge pain in the butt, I wish someone could solve this". If there's a problem, usually there is just one solution that everyone adopts, builds on and improves (which is ironic considering Perl's motto) Whereas with Python, if there's a problem, someone tries to solve it, but it's not perfect, so somebody makes a second solution, then a third, fourth, fifth, etc. Python never established the convention of reusing and improving existing things. Python has a forking/independent culture. So there will always be 20 different ways to do something and none of them all that great (which is ironic considering Python's principles)
deleted 2y ago
[deleted]
formerly_proven 2y ago
Most other packaging systems don't even attempt to solve the problems that e.g. wheels solve.
mjw1007 2y ago
The early days of setuptools/easy_install (around 2004) were interesting. The way I remember it, the main developer had a clear "don't ask for permission, just build what you think is best and put it out there" approach. That avoids the problems you can get where everything gets discussed to death and nothing actually gets written, but the downside was that the project spent a long time in an uneasy situation where it didn't have complete official buy-in but was popular enough that it would have been hard to push an alternative. There were a couple of questionable early design decisions that didn't help.

wruza 2y ago

There’s also no package caching across venvs(?). Every time I have to install something AI it’s the same 2GB torch + few hundred megabytes libs + cuda update of downloads again and again. On top of that, python projects love to start downloading large resources without asking, and if you ctrl-c it, the setup process is usually non-restartable. You’re just left with a broken folder that doesn’t run anymore. Feels like everything about python’s “install” is an afterthought at best.

kzrdude 2y ago
uv by astral is cleaning up and improving a lot in this area right now
- nubinetwork 2y ago
  Pip, conda, uv... how many package managers do we need?
  - IanCal 2y ago
    Just one, but one that's decent. We'll keep inventing new ones until then. Maybe beyond.
    - cdchn 2y ago
      I don't understand the obsession with reimplementing Python utilities or ecosystem elements in Rust.
  - kzrdude 2y ago
    Uv's adoption rate says that there was a huge demand for it
  - maleldil 2y ago
    uv is essentially an improved version of pip, virtualenv and pip-tools, but from the outside it's pretty much the same thing. The CLI imitates most of pip's options too. It's not like conda, which is a completely different ecosystem, with dodgy integration with pip.

senkora 2y ago

> To allow for compact filenames of bdists that work with more than one compatibility tag triple, each tag in a filename can instead be a ‘.’-separated, sorted, set of tags. Is his second point about compressed tag sets right? I would interpret "sorted, set" to disallow unsorted and duplicate tags.

woodruffw 2y ago
Two aspects: 1. Sorted, set is a contraction in terms: `packaging` and other tooling have interpreted it to mean just "set," which implies unsorted. 2. The quoted language there isn't from the wheel filename definition itself (and isn't linked to it), so it's unclear whether it's intended to be normative. Similarly, "can" is weaker than "should."

amelius 2y ago

Also, when installing a wheel you can get a brief error message "Not a supported wheel on this platform" without an explanation of why this wheel is not compatible with your system, which can be infuriating.

zbentley 2y ago
Yeah, that's incredibly frustrating. There's an open issue about it which you can express support for, and some marginally-hacky code that you can paste into a "debug_wheel.py"-ish script to get unblocked in the short term, here: https://github.com/pypa/pip/issues/10793 https://github.com/pypa/pip/issues/10793

ethbr1 2y ago

I assumed by now that "Python _____ has no canonical form" was a feature, not a bug, to the Python community. Let them eat cake!

deleted 2y ago
[deleted]
deleted 2y ago
[deleted]

agumonkey 2y ago

Not helped by every boundaries you'll have to cross to deploy a python lib or app. Pypa will not necessarily agree with pypi and or docker image versionning. It's amazing how fast naming hell travels.

nerdponx 2y ago

Note to readers who enjoy complaining about Python: the title is more bad-sounding than the actual content. It's analogous to how IDNs and IP addresses have no single canonical form, but 2 or more equivalent forms that can be parsed. To properly compare two IP addresses, you must parse them and compare the results. The same thing is happening here. It's an interesting problem and maybe an oversight in the design of the wheel name format, but it's not really an instance of "Python bad" the way you might assume based on the title.

woodruffw 2y ago
Right -- I'd consider it strictly a curiosity, and not even a flaw.

zbentley 2y ago

In addition to standardization headaches, the wheel issue discussed is also hindered by using file names as an at least somewhat complex medatata structure's transport. Sure, you can do that, but it's a pain in the ass even if you do a good job. A file name is just one string, after all. This is a somewhat hot take, but I wish there were more widespread use of package repository tools that don't use the dpkg model of "the registry's metadata is baked into a simple file hierarchy". The simplicity and accessibility (I can just browse packages with HTTP/FTP) of that approach is appealing, but it breaks down a lot in cases like this. I don't think it's too opaque or overcomplicated to wish for repository servers that have a pair of access APIs: one to ask the server about metadata/packages matching a set of constraints, and another to download blobs/tarballs/whatever. Docker takes that approach, and it seems to work well. I just wish that instances of this pattern weren't per-language/platform and we had something a little more standardized.

VagabundoP 2y ago

I've been testing out and migrating my hobby projects to Rye/uv and while they are still buggy in some cases i've been pleased with the results. There's still work to do but Rye works well with the standards and conventions as they currently exist.