4 ms·

The pypa maintainers, in my experience, will engage with such issues but ultimately discussion dies off and they simply ignore corner cases. Realistically speak

The pypa maintainers, in my experience, will engage with such issues but ultimately discussion dies off and they simply ignore corner cases. Realistically speaking if you want to ship impure wheels for anything outside of ppc(64le), armhf, x86 or amd64 you're SOL. Python packaging is an absolute shit show especially where impure wheels are involved. Between the manylinux "standards" that require downloading random Centos docker images because their libc is "probably old enough" to the setuptools vs distutils vs whatever else with none of them having actually mature support for binary extensions (and even less - really zero, and no docs - for raw .so's via ctypes) to the byzantine PyPI upload procedures with its underspecified platform tags (really a manylinux problem though), my Python packaging experience was so bad that I ultimately gave up on the Python ecosystem as a whole. Shipping applications and libraries with this language is simply awful to the point where I'd rather use a compiled language because it is easier to build binaries for 10 different arches under qemu than it is to ship a fucking wheel. I'm so over Python.

geofft 6y ago
Hmm. Why can't you build wheels for 10 different arches with your same qemu setup? Is it that PyPI won't actually accept those wheels and so you'd have to self-host? (I have contributed a tiny bit to the manylinux stuff, so I'm curious where the problems are, but I'm definitely not active in any sense... it does seem like a couple of these issues boil down to "there should be more people spending time on it," sadly.)
- corndoge 6y ago
  > Why can't you build wheels for 10 different arches with your same qemu setup? This gets pretty difficult when you are required to do it in Docker in order to produce manylinux-compatible wheels. Or at least that is my understanding. For some of the arches I need to support I just do it on real hardware - but of course the quay.io manylinux images aren't available for those arches. It is a half baked solution. > Is it that PyPI won't actually accept those wheels and so you'd have to self-host? Yeah, that's basically what it boils down to. The platform tags are underspecified - especially for ARM, though this is hardly a python specific problem since few people seem to really understand what the differences are in ARM ABIs - but PyPI accepts a subset of even those, which is limiting. The other difficulty is that the packaging infrastructure for impure wheels that aren't strictly cpython extensions is basically nonexistent. The packaging procedure for a module that imports a .so via ctypes, for example, is basically "good luck". To digress slightly, another problem is that source distributions for such packages so frequently throw some obscure python stack trace or a "lol no gcc" stack trace when you try to install them in pip. Which to me is an unacceptable way to communicate a problem. Pip's apparent worldview that tracebacks are an okay way to communicate errors is so hostile to end users, and I see that same attitude present throughout the Python ecosystem. I mean the whole concept of venv's as standard procedure is a good indicator the packaging system is fubar. Sorry for the rant.
aaronbrethorst 6y ago
I have absolutely no idea what you are talking about, and it makes me so happy that I blundered into Ruby in 2007. For comparison, with Ruby you use RVM or Rbenv plus a Gemfile with Bundler and you’re done. There’s no debate to speak of. The seemingly equivalent nightmare that I see in JavaScript inspires me to go use Ember if only because its creator was instrumental in forming the modern Ruby tooling ecosystem, and I hear that ember is equally unlikely to make me want to stab my forehead with a fork in frustration.
- corndoge 6y ago
  How does Ruby handle C extensions?
  - aaronbrethorst 6y ago
    Well enough for my needs on a Mac and deploying to Heroku.
- sirn 6y ago
  RubyGem doesn't do binary distribution of C extension at all. Gem with C extension is usually built at install time (via extconf/mkmf) which side-stepped the whole shipping .so for 10 different arches issue but it create few other issues of requiring build tool to be installed on production server to install a Gem (so you're back to the Python's manylinux issue, but applies to all arches instead of only edge cases), and any Gem that need to use pre-built .so need to do it themselves (including differentiating between arches).
  - aaronbrethorst 6y ago
    RubyGem doesn't do binary distribution of C extension at all Judging by what I’ve read here and experienced firsthand with the state of python tooling, this seems like it can be a worthwhile trade off
    - sirn 6y ago
      Old-style Python sdist/egg is exactly the same and it has the two issues mentioned in my original reply. Python sdist/Ruby extconf/mkmf also has an issue of executing code from the package itself to install a package and can be a huge security issue (which Python wheels were supposed to solve). The reason this has never become a problem in Ruby world is because Ruby community is centered around Rails, of which a requirement to deploy on non-x86_64/ARM64 is pretty much non-existent. Ruby itself doesn't even support non-glibc (e.g. compiling Ruby on musl requires a patch).

geofft 6y ago

Hmm. Why can't you build wheels for 10 different arches with your same qemu setup? Is it that PyPI won't actually accept those wheels and so you'd have to self-host? (I have contributed a tiny bit to the manylinux stuff, so I'm curious where the problems are, but I'm definitely not active in any sense... it does seem like a couple of these issues boil down to "there should be more people spending time on it," sadly.)

corndoge 6y ago
> Why can't you build wheels for 10 different arches with your same qemu setup? This gets pretty difficult when you are required to do it in Docker in order to produce manylinux-compatible wheels. Or at least that is my understanding. For some of the arches I need to support I just do it on real hardware - but of course the quay.io manylinux images aren't available for those arches. It is a half baked solution. > Is it that PyPI won't actually accept those wheels and so you'd have to self-host? Yeah, that's basically what it boils down to. The platform tags are underspecified - especially for ARM, though this is hardly a python specific problem since few people seem to really understand what the differences are in ARM ABIs - but PyPI accepts a subset of even those, which is limiting. The other difficulty is that the packaging infrastructure for impure wheels that aren't strictly cpython extensions is basically nonexistent. The packaging procedure for a module that imports a .so via ctypes, for example, is basically "good luck". To digress slightly, another problem is that source distributions for such packages so frequently throw some obscure python stack trace or a "lol no gcc" stack trace when you try to install them in pip. Which to me is an unacceptable way to communicate a problem. Pip's apparent worldview that tracebacks are an okay way to communicate errors is so hostile to end users, and I see that same attitude present throughout the Python ecosystem. I mean the whole concept of venv's as standard procedure is a good indicator the packaging system is fubar. Sorry for the rant.

aaronbrethorst 6y ago

I have absolutely no idea what you are talking about, and it makes me so happy that I blundered into Ruby in 2007. For comparison, with Ruby you use RVM or Rbenv plus a Gemfile with Bundler and you’re done. There’s no debate to speak of. The seemingly equivalent nightmare that I see in JavaScript inspires me to go use Ember if only because its creator was instrumental in forming the modern Ruby tooling ecosystem, and I hear that ember is equally unlikely to make me want to stab my forehead with a fork in frustration.

corndoge 6y ago
How does Ruby handle C extensions?
- aaronbrethorst 6y ago
  Well enough for my needs on a Mac and deploying to Heroku.
sirn 6y ago
RubyGem doesn't do binary distribution of C extension at all. Gem with C extension is usually built at install time (via extconf/mkmf) which side-stepped the whole shipping .so for 10 different arches issue but it create few other issues of requiring build tool to be installed on production server to install a Gem (so you're back to the Python's manylinux issue, but applies to all arches instead of only edge cases), and any Gem that need to use pre-built .so need to do it themselves (including differentiating between arches).
- aaronbrethorst 6y ago
  RubyGem doesn't do binary distribution of C extension at all Judging by what I’ve read here and experienced firsthand with the state of python tooling, this seems like it can be a worthwhile trade off
  - sirn 6y ago
    Old-style Python sdist/egg is exactly the same and it has the two issues mentioned in my original reply. Python sdist/Ruby extconf/mkmf also has an issue of executing code from the package itself to install a package and can be a huge security issue (which Python wheels were supposed to solve). The reason this has never become a problem in Ruby world is because Ruby community is centered around Rails, of which a requirement to deploy on non-x86_64/ARM64 is pretty much non-existent. Ruby itself doesn't even support non-glibc (e.g. compiling Ruby on musl requires a patch).