3 ms·

Agree. I had a build pipeline pulling down an internal package to build into a `.pex` [0] file for running a Python program in a release pipeline that did not h

Agree. I had a build pipeline pulling down an internal package to build into a `.pex` [0] file for running a Python program in a release pipeline that did not have access to the internal package repository. I came up with this solution because the typical virtualenv that you might build locally does not appreciate being copied wholesale to another system (iirc it had some absolute paths). One day the release pipeline broke because a dependency's authors published a new version containing a wheel for a newer ABI, though the older ABI was still supported. My build pipeline pulled the one for the later ABI because it was compatible, but the release pipeline environment, which could not be upgraded, did not have the newer version of gcc required. It was a nasty introduction to the fact that one Python package version can have multiple published wheels targeting different C++ compilers! But the worst part was not that I learned something new! It was that I wrangled with both pex and pip for a few hours trying to figure out how to download a wheel for an older ABI as both tools resolve the latest compatible by default. The options to do so are ostensibly there but the effect was not. I definitely could have made some mistakes but in my mind it shouldn't be _hard_ to get it right. I ended up downloading the wheel for the internal package in the build and copying it directly into the artifact. The release pipeline resolved public dependencies and installed the internal package from the local file. [0] - https://pex.readthedocs.io/en/latest/ https://pex.readthedocs.io/en/latest/

orf 4y ago
The release pipeline broke because you didn't lock your dependencies.
- noisenotsignal 4y ago
  Not sure what you mean. pex is the dependency lock, at least from a version perspective. To be clear, it broke after I intentionally upgraded the offending package to the later version. I admit to being unaware of "versions inside versions" where a version may have multiple published ABIs that are not compatible across systems, but a nice packaging system would still make it easy for me to use platform X to build for platform Y.
  - orf 4y ago
    Oh, from your "One day the release pipeline broke because a dependency's authors published a new version..." wording it seemed that this breakage occurred without you upgrading anything.
    - noisenotsignal 4y ago
      Sorry for the wording, I see how it can be interpreted that way. Seems I skipped a few details on accident.
bsder 4y ago
> It was a nasty introduction to the fact that one Python package version can have multiple published wheels targeting different C++ compilers! Erm, is this not the failure of C++ standards instead of Python? This seems like Python is the failure point being caused by C++ not having a stable ABI. Python is simply trying to paper over the brokenness of the C++ ecosystem. Am I missing something?
- noisenotsignal 4y ago
  That’s a good point! However, there’s still things Python could do to help make the experience better. The package I was using had multiple artifacts per version, including the raw zipped source code and wheels for each ABI they built for. It’s certainly convenient that the most applicable one for your current system is pulled automatically by pip. But in this case that’s not the behavior I wanted, and I could not successfully get pex or pip to download a “less optimal” artifact. You can blame C++, but if Python is already papering over deficiencies it’s not unreasonable to expect improvements that aren’t fundamentally impossible. It should definitely be possible for there to be one easily understood argument to specify which artifact pip should download. In my (potentially flawed) experience, figuring out which ABI value to pass was already not straightforward. Even then, the arguments for selecting an ABI or even asking for the raw source instead of a wheel weren’t seeming to have an effect.
- inferiorhuman 4y ago
  Python is simply trying to paper over the brokenness of the C++ ecosystem. I'd be pretty comfortable saying that this is indeed a Python problem. C++ ABI interop has been an issue for decades. Instead of papering over it, Python should be exposing a host triplet (or whatever) so that end users can more easily identify what is and isn't compatible. Pretty much every other ecosystem (except perhaps Javascript?) does this. Look at FreeBSD packages, the ABI is used as part of the identifier. GNU Autoconf, rvm, homebrew, rust, etc., etc. all use and expose an ABI identifier so that you aren't accidentally going to mix and match things.
- structural 4y ago
  Python packaging doesn't have good ways to depend on specific versions of <insert-other-language-here> packages. This isn't specific to C/C++ and is mostly historical: it's designed for an ecosystem of packages where everything is written in Python! But nowadays, more and more Python libraries are bindings to code written in many other languages, so you need a package management approach that includes this.

orf 4y ago

The release pipeline broke because you didn't lock your dependencies.

noisenotsignal 4y ago
Not sure what you mean. pex is the dependency lock, at least from a version perspective. To be clear, it broke after I intentionally upgraded the offending package to the later version. I admit to being unaware of "versions inside versions" where a version may have multiple published ABIs that are not compatible across systems, but a nice packaging system would still make it easy for me to use platform X to build for platform Y.
- orf 4y ago
  Oh, from your "One day the release pipeline broke because a dependency's authors published a new version..." wording it seemed that this breakage occurred without you upgrading anything.
  - noisenotsignal 4y ago
    Sorry for the wording, I see how it can be interpreted that way. Seems I skipped a few details on accident.

bsder 4y ago

> It was a nasty introduction to the fact that one Python package version can have multiple published wheels targeting different C++ compilers! Erm, is this not the failure of C++ standards instead of Python? This seems like Python is the failure point being caused by C++ not having a stable ABI. Python is simply trying to paper over the brokenness of the C++ ecosystem. Am I missing something?

noisenotsignal 4y ago
That’s a good point! However, there’s still things Python could do to help make the experience better. The package I was using had multiple artifacts per version, including the raw zipped source code and wheels for each ABI they built for. It’s certainly convenient that the most applicable one for your current system is pulled automatically by pip. But in this case that’s not the behavior I wanted, and I could not successfully get pex or pip to download a “less optimal” artifact. You can blame C++, but if Python is already papering over deficiencies it’s not unreasonable to expect improvements that aren’t fundamentally impossible. It should definitely be possible for there to be one easily understood argument to specify which artifact pip should download. In my (potentially flawed) experience, figuring out which ABI value to pass was already not straightforward. Even then, the arguments for selecting an ABI or even asking for the raw source instead of a wheel weren’t seeming to have an effect.
inferiorhuman 4y ago
Python is simply trying to paper over the brokenness of the C++ ecosystem. I'd be pretty comfortable saying that this is indeed a Python problem. C++ ABI interop has been an issue for decades. Instead of papering over it, Python should be exposing a host triplet (or whatever) so that end users can more easily identify what is and isn't compatible. Pretty much every other ecosystem (except perhaps Javascript?) does this. Look at FreeBSD packages, the ABI is used as part of the identifier. GNU Autoconf, rvm, homebrew, rust, etc., etc. all use and expose an ABI identifier so that you aren't accidentally going to mix and match things.
structural 4y ago
Python packaging doesn't have good ways to depend on specific versions of <insert-other-language-here> packages. This isn't specific to C/C++ and is mostly historical: it's designed for an ecosystem of packages where everything is written in Python! But nowadays, more and more Python libraries are bindings to code written in many other languages, so you need a package management approach that includes this.