5 ms·
Quirks of Python package versioning
- woodruffw 3y agoThis is a great summary -- epoch versioning in particular is a useful and un(der)used feature in PEP 440 versioning!
- dmbaggett 3y agoUnfortunately it's not perfectly supported. If you set up a private registry on gitlab and try to push packages with epoch versions, gitlab will normalize the ! to _ so everything breaks. (We found this out the hard way recently.) Another Python versioning gotcha I've never see discussed is use of instructions like AVX2 in C extensions. There seems to be no way in Python versioning to say "this version is for CPUs with AVX2 instructions" so I guess public C extensions have to be compiled for the least common denominator instruction set. Does anyone know otherwise? How do other packaging systems like cargo deal with this?
- woodruffw 3y ago> If you set up a private registry on gitlab and try to push packages with epoch versions, gitlab will normalize the ! to _ so everything breaks. (We found this out the hard way recently.) That's unfortunate; I know Poetry's support for epochs has also been spotty (at least historically, I'm not sure about currently). > There seems to be no way in Python versioning to say "this version is for CPUs with AVX2 instructions" so I guess public C extensions have to be compiled for the least common denominator instruction set. Does anyone know otherwise? This is my understanding as well: x86_64 in wheel tags implies the lowest common denominator for AMD64. I think the best you can generally do here is either (1) accept the pain that comes with sdists, or (2) do runtime feature detection for things like AVX2. > How do other packaging systems like cargo deal with this? Cargo always builds from source, so they effectively punt on this problem.
- mxmlnkn 3y agoOr simply require an AVX2-capable CPU. According to Steam survey, 90% have AVX2 support. That's not as nice as the 99.18% SSE4.2 support or the 100% SSE3 support, but for some use cases such as scientific libraries, requiring a CPU that is less than 8 years old makes sense to me. Some more helpful error message than "Illegal Instruction" would still be a good idea though. This does require runtime AVX2 detection but at least it avoids the pain of trying to compile "pristine" architecture-specific code, which seems only possible for C/C++ code that does not use any standard library function or by writing assembler directly.
- appplication 3y ago> Instead use the official packaging.version.Version class to parse, compare, and reason about Python versions. I would have agreed with this previously, but a year or so ago they changed their backend and now it just errors out on certain (maybe not common, but extant) version strings, such as cloudera’s spark release version. > packaging.version.InvalidVersion: Invalid version: '2.4.0.cloudera2' If I recall, it used to parse fine on their old backend parser. Sure, the maintainers can cast aspersions on uneducated devs from their ivory tower, amazed anyone would think this is a valid version. But things like this can and do show up in the wild, and the parsing package playing dumb (especially when it didn’t use to) doesn’t inspire a lot of confidence that this should be the tool you use. Just parse the damn thing. It’s painfully obvious how to do so. I should have to write my own custom parser to wrap theirs. Sure, it’s nonstandard, but it should be handled as a best guess + a warning, not an exception. I didn’t choose this version string. Don’t punish me for trying to parse it.
- woodruffw 3y ago"Just parse the damn thing" without a grammar leads to exploitable parser differentials[1]. Your "obvious" parse isn't everyone else's. (Nobody anywhere has called developers "dumb" for using legacy, non-PEP 440 versions. There's nothing else they even could have done prior to version standardization across the Python ecosystem.) [1]: https://blog.yossarian.net/2022/05/09/A-most-vexing-parse-but-for-Python-packaging https://blog.yossarian.net/2022/05/09/A-most-vexing-parse-bu...
- appplication 3y agoI understand that the lack of standardization makes things difficult, but there are a lot of knowables. Further, there would be nothing wrong with returning something like a NonconformingVersion object, where we can best guess at things like major, minor, patch. Parse what you think you know, and when you run into something that is nonconforming, then you fail over to return an object that has what you think you know and what you can’t sort out. E.g: in the above example, major, minor, patch versions are apparent. I would even say the vast majority of nonconforming versions have at least one or more of major/minor/patch readily parseable, the problem is we aren’t sure exactly how of handle the rest of it. And that’s fine. This approach is super pythonic in the BTAFTP philosophy, where I will let users decide what to do with the object returned. “You asked me to parse this mess. I can’t fully do that, but here’s a best guess.” This is worlds better than just failing completely.
- RazerM 3y ago> to parse, compare, and reason about Python versions. If in addition to the above you want to mutate Python versions, you may consider my parver package[1]. It also doesn't normalise the input unless you ask it to, so you can preserve your preferred format after mutating. [1]: https://github.com/RazerM/parver https://github.com/RazerM/parver
- djha-skin 3y agoI wrote a Clojure library that implemented version comparison for all of the package versioning algorithms I could find[1], and by faaar the Python version comparison algorithm was the weirdest, most annoying one to implement (PEP 440[2], the current standard at the time). So many exceptions to the rules that attempt to be helpful but didn't really help. In my opinion, the best version comparison algorithm is that of the Debian package manager. Super simple, and very flexible. The description of the algorithm fits in just three paragraphs[3]. Most other algorithms implemented in the aforementioned library were actually implemented in terms of it. Prereleases, fourth version numbers, etc. anything you want basically fits nicely in that box. PEP 440 was the only version comparison algorithm that couldn't be neatly implemented in terms of that of Debian's, beating out semver, maven, and rubygems for complexity. 1: http://djhaskin987.gitlab.io/serovers/index.html http://djhaskin987.gitlab.io/serovers/index.html 2: https://peps.python.org/pep-0440/ https://peps.python.org/pep-0440/ 3: https://www.debian.org/doc/debian-policy/ch-controlfields.html#version https://www.debian.org/doc/debian-policy/ch-controlfields.ht...
- woodruffw 3y agoPEP 440 is complicated, but it's also complicated in ways that make sense for a language packaging ecosystem: Debian is tasked primarily with distributing stable builds of software; PyPI is facilitating a source and build distribution relationship between different maintainers and users. In other words: Debian prioritizes unified versioning and has a "fan-out" topology of users; PyPI prioritizes facilitating dependency resolution for a "graph" topology of users.
- djha-skin 3y agoTrue, but I'm also talking about semver, rubygems, and maven algorithms, all of which are also used for languages. It may have been designed for that use case, but so were these others. They also had simpler designs.
- jihadjihad 3y agoOT, but it took me an embarrassing amount of time before I realized that '5.5.1' != '5.5.10'. Maybe it's the fact that there isn't a leading zero, like .01, or something, but it was a moment of "wait...did I know that? Is that right?"
- nneonneo 3y agoTrivia: the package with the longest version number is uselesscapitalquiz (https://pypi.org/project/uselesscapitalquiz https://pypi.org/project/uselesscapitalquiz), version 3.14159265358979323846264338327950288419716939937510582097494459230781640628620899862803482534211706798214808651328230664709384460955058223172535940812848111745028410270193852110555964462294895493038196442881097566593 This is mentioned in the post, but the actual package was not identified.