3 ms·

I'm a big fan of Poetry (I have a few commits in there) but it's not without its downsides. At least a few months ago, you needed to install an entire compilat

I'm a big fan of Poetry (I have a few commits in there) but it's not without its downsides. At least a few months ago, you needed to install an entire compilation suite if you wanted to install a pure-Python source distribution (sdist) that was created with Poetry on Alpine. Somewhat by design, sdists created with Poetry require Poetry to install. Poetry itself is dependent on the cryptography package which, like all like all packages that use native-C extensions, suffers from a lack of musl-compatible wheels.

geofft 6y ago
Hmm, this is solvable in theory by defining a platform tag for musl such that authors of C extensions could upload musl wheels, right? I see some discussion at https://github.com/pypa/manylinux/issues/37 https://github.com/pypa/manylinux/issues/37 but I don't totally follow why it died off.
- deleted 6y ago
  [deleted]
- Caligatio 6y ago
  You can really go down the rabbit hole trying to figure out where the wheel+Alpine problem should be fixed. https://github.com/pypa/packaging-problems/issues/69 https://github.com/pypa/packaging-problems/issues/69 is a newer bug that is tracking this. Python relies on the "manylinux" specification (https://github.com/pypa/manylinux https://github.com/pypa/manylinux) which originally pinned build environments to CentOS 5, was updated in 2018 to use CentOS 6, and then updated in 2019 to use CentOS 7. It's entirely possible that the Alpine user base isn't as big/important as I would think but this seems like something that needs to be solved.
- corndoge 6y ago
  The pypa maintainers, in my experience, will engage with such issues but ultimately discussion dies off and they simply ignore corner cases. Realistically speaking if you want to ship impure wheels for anything outside of ppc(64le), armhf, x86 or amd64 you're SOL. Python packaging is an absolute shit show especially where impure wheels are involved. Between the manylinux "standards" that require downloading random Centos docker images because their libc is "probably old enough" to the setuptools vs distutils vs whatever else with none of them having actually mature support for binary extensions (and even less - really zero, and no docs - for raw .so's via ctypes) to the byzantine PyPI upload procedures with its underspecified platform tags (really a manylinux problem though), my Python packaging experience was so bad that I ultimately gave up on the Python ecosystem as a whole. Shipping applications and libraries with this language is simply awful to the point where I'd rather use a compiled language because it is easier to build binaries for 10 different arches under qemu than it is to ship a fucking wheel. I'm so over Python.
  - geofft 6y ago
    Hmm. Why can't you build wheels for 10 different arches with your same qemu setup? Is it that PyPI won't actually accept those wheels and so you'd have to self-host? (I have contributed a tiny bit to the manylinux stuff, so I'm curious where the problems are, but I'm definitely not active in any sense... it does seem like a couple of these issues boil down to "there should be more people spending time on it," sadly.)
    - corndoge 6y ago
      > Why can't you build wheels for 10 different arches with your same qemu setup? This gets pretty difficult when you are required to do it in Docker in order to produce manylinux-compatible wheels. Or at least that is my understanding. For some of the arches I need to support I just do it on real hardware - but of course the quay.io manylinux images aren't available for those arches. It is a half baked solution. > Is it that PyPI won't actually accept those wheels and so you'd have to self-host? Yeah, that's basically what it boils down to. The platform tags are underspecified - especially for ARM, though this is hardly a python specific problem since few people seem to really understand what the differences are in ARM ABIs - but PyPI accepts a subset of even those, which is limiting. The other difficulty is that the packaging infrastructure for impure wheels that aren't strictly cpython extensions is basically nonexistent. The packaging procedure for a module that imports a .so via ctypes, for example, is basically "good luck". To digress slightly, another problem is that source distributions for such packages so frequently throw some obscure python stack trace or a "lol no gcc" stack trace when you try to install them in pip. Which to me is an unacceptable way to communicate a problem. Pip's apparent worldview that tracebacks are an okay way to communicate errors is so hostile to end users, and I see that same attitude present throughout the Python ecosystem. I mean the whole concept of venv's as standard procedure is a good indicator the packaging system is fubar. Sorry for the rant.
  - aaronbrethorst 6y ago
    I have absolutely no idea what you are talking about, and it makes me so happy that I blundered into Ruby in 2007. For comparison, with Ruby you use RVM or Rbenv plus a Gemfile with Bundler and you’re done. There’s no debate to speak of. The seemingly equivalent nightmare that I see in JavaScript inspires me to go use Ember if only because its creator was instrumental in forming the modern Ruby tooling ecosystem, and I hear that ember is equally unlikely to make me want to stab my forehead with a fork in frustration.

geofft 6y ago

Hmm, this is solvable in theory by defining a platform tag for musl such that authors of C extensions could upload musl wheels, right? I see some discussion at https://github.com/pypa/manylinux/issues/37 https://github.com/pypa/manylinux/issues/37 but I don't totally follow why it died off.

deleted 6y ago
[deleted]
Caligatio 6y ago
You can really go down the rabbit hole trying to figure out where the wheel+Alpine problem should be fixed. https://github.com/pypa/packaging-problems/issues/69 https://github.com/pypa/packaging-problems/issues/69 is a newer bug that is tracking this. Python relies on the "manylinux" specification (https://github.com/pypa/manylinux https://github.com/pypa/manylinux) which originally pinned build environments to CentOS 5, was updated in 2018 to use CentOS 6, and then updated in 2019 to use CentOS 7. It's entirely possible that the Alpine user base isn't as big/important as I would think but this seems like something that needs to be solved.
corndoge 6y ago
The pypa maintainers, in my experience, will engage with such issues but ultimately discussion dies off and they simply ignore corner cases. Realistically speaking if you want to ship impure wheels for anything outside of ppc(64le), armhf, x86 or amd64 you're SOL. Python packaging is an absolute shit show especially where impure wheels are involved. Between the manylinux "standards" that require downloading random Centos docker images because their libc is "probably old enough" to the setuptools vs distutils vs whatever else with none of them having actually mature support for binary extensions (and even less - really zero, and no docs - for raw .so's via ctypes) to the byzantine PyPI upload procedures with its underspecified platform tags (really a manylinux problem though), my Python packaging experience was so bad that I ultimately gave up on the Python ecosystem as a whole. Shipping applications and libraries with this language is simply awful to the point where I'd rather use a compiled language because it is easier to build binaries for 10 different arches under qemu than it is to ship a fucking wheel. I'm so over Python.
- geofft 6y ago
  Hmm. Why can't you build wheels for 10 different arches with your same qemu setup? Is it that PyPI won't actually accept those wheels and so you'd have to self-host? (I have contributed a tiny bit to the manylinux stuff, so I'm curious where the problems are, but I'm definitely not active in any sense... it does seem like a couple of these issues boil down to "there should be more people spending time on it," sadly.)
  - corndoge 6y ago
    > Why can't you build wheels for 10 different arches with your same qemu setup? This gets pretty difficult when you are required to do it in Docker in order to produce manylinux-compatible wheels. Or at least that is my understanding. For some of the arches I need to support I just do it on real hardware - but of course the quay.io manylinux images aren't available for those arches. It is a half baked solution. > Is it that PyPI won't actually accept those wheels and so you'd have to self-host? Yeah, that's basically what it boils down to. The platform tags are underspecified - especially for ARM, though this is hardly a python specific problem since few people seem to really understand what the differences are in ARM ABIs - but PyPI accepts a subset of even those, which is limiting. The other difficulty is that the packaging infrastructure for impure wheels that aren't strictly cpython extensions is basically nonexistent. The packaging procedure for a module that imports a .so via ctypes, for example, is basically "good luck". To digress slightly, another problem is that source distributions for such packages so frequently throw some obscure python stack trace or a "lol no gcc" stack trace when you try to install them in pip. Which to me is an unacceptable way to communicate a problem. Pip's apparent worldview that tracebacks are an okay way to communicate errors is so hostile to end users, and I see that same attitude present throughout the Python ecosystem. I mean the whole concept of venv's as standard procedure is a good indicator the packaging system is fubar. Sorry for the rant.
- aaronbrethorst 6y ago
  I have absolutely no idea what you are talking about, and it makes me so happy that I blundered into Ruby in 2007. For comparison, with Ruby you use RVM or Rbenv plus a Gemfile with Bundler and you’re done. There’s no debate to speak of. The seemingly equivalent nightmare that I see in JavaScript inspires me to go use Ember if only because its creator was instrumental in forming the modern Ruby tooling ecosystem, and I hear that ember is equally unlikely to make me want to stab my forehead with a fork in frustration.