4 ms·

> It was a nasty introduction to the fact that one Python package version can have multiple published wheels targeting different C++ compilers! Erm, is this no

by bsder 4y ago

> It was a nasty introduction to the fact that one Python package version can have multiple published wheels targeting different C++ compilers! Erm, is this not the failure of C++ standards instead of Python? This seems like Python is the failure point being caused by C++ not having a stable ABI. Python is simply trying to paper over the brokenness of the C++ ecosystem. Am I missing something?

noisenotsignal 4y ago
That’s a good point! However, there’s still things Python could do to help make the experience better. The package I was using had multiple artifacts per version, including the raw zipped source code and wheels for each ABI they built for. It’s certainly convenient that the most applicable one for your current system is pulled automatically by pip. But in this case that’s not the behavior I wanted, and I could not successfully get pex or pip to download a “less optimal” artifact. You can blame C++, but if Python is already papering over deficiencies it’s not unreasonable to expect improvements that aren’t fundamentally impossible. It should definitely be possible for there to be one easily understood argument to specify which artifact pip should download. In my (potentially flawed) experience, figuring out which ABI value to pass was already not straightforward. Even then, the arguments for selecting an ABI or even asking for the raw source instead of a wheel weren’t seeming to have an effect.
inferiorhuman 4y ago
Python is simply trying to paper over the brokenness of the C++ ecosystem. I'd be pretty comfortable saying that this is indeed a Python problem. C++ ABI interop has been an issue for decades. Instead of papering over it, Python should be exposing a host triplet (or whatever) so that end users can more easily identify what is and isn't compatible. Pretty much every other ecosystem (except perhaps Javascript?) does this. Look at FreeBSD packages, the ABI is used as part of the identifier. GNU Autoconf, rvm, homebrew, rust, etc., etc. all use and expose an ABI identifier so that you aren't accidentally going to mix and match things.
structural 4y ago
Python packaging doesn't have good ways to depend on specific versions of <insert-other-language-here> packages. This isn't specific to C/C++ and is mostly historical: it's designed for an ecosystem of packages where everything is written in Python! But nowadays, more and more Python libraries are bindings to code written in many other languages, so you need a package management approach that includes this.
- humanrebar 4y ago
  There's no way to depend on specific versions of C or C++. The actual binaries vary depending on architecture, compilation toolchain settings, compilation flags, link options, and all sorts of other things. There's really no way to address those built artifacts even if they were available on pypi or something.
  - Filligree 4y ago
    Well, there is: It's Nix's buildPythonApplication. Trying to use Python in Nix is frequently a bother, but at least once it works, it works.
  - structural 4y ago
    There's a couple different ways. 1. Build the artifact that you depend on, publish it on "pypi or something" referenced by both the version of the library and a hash that uniquely identifies the build. This is the approach taken by conda, for better or worse. 2. Only allow one canonical build of a library you depend on so that the "version" becomes a unique identifier for the build. This is the approach taken by distribution package managers. 3. Create metapackages that describe ABI constraints that are required by packages and must be satisfied by the underlying system. For example, a "cxxabi" package could be provided by the underlying system, and the packaging tools could automatically add dependencies on "cxxabi" at build time, based on either an exact pin to the library built against, or in some cases, a relaxed dependency by inspection of versioned symbols used by the binary. 4. Statically link all your dependencies and/or vendor all your dependencies. These are used by quite a few pypi packages that depend on standalone C libraries to avoid most of the issues altogether.

by bsder 4y ago

noisenotsignal 4y ago

That’s a good point! However, there’s still things Python could do to help make the experience better. The package I was using had multiple artifacts per version, including the raw zipped source code and wheels for each ABI they built for. It’s certainly convenient that the most applicable one for your current system is pulled automatically by pip. But in this case that’s not the behavior I wanted, and I could not successfully get pex or pip to download a “less optimal” artifact. You can blame C++, but if Python is already papering over deficiencies it’s not unreasonable to expect improvements that aren’t fundamentally impossible. It should definitely be possible for there to be one easily understood argument to specify which artifact pip should download. In my (potentially flawed) experience, figuring out which ABI value to pass was already not straightforward. Even then, the arguments for selecting an ABI or even asking for the raw source instead of a wheel weren’t seeming to have an effect.

inferiorhuman 4y ago

Python is simply trying to paper over the brokenness of the C++ ecosystem. I'd be pretty comfortable saying that this is indeed a Python problem. C++ ABI interop has been an issue for decades. Instead of papering over it, Python should be exposing a host triplet (or whatever) so that end users can more easily identify what is and isn't compatible. Pretty much every other ecosystem (except perhaps Javascript?) does this. Look at FreeBSD packages, the ABI is used as part of the identifier. GNU Autoconf, rvm, homebrew, rust, etc., etc. all use and expose an ABI identifier so that you aren't accidentally going to mix and match things.

structural 4y ago

Python packaging doesn't have good ways to depend on specific versions of <insert-other-language-here> packages. This isn't specific to C/C++ and is mostly historical: it's designed for an ecosystem of packages where everything is written in Python! But nowadays, more and more Python libraries are bindings to code written in many other languages, so you need a package management approach that includes this.

humanrebar 4y ago
There's no way to depend on specific versions of C or C++. The actual binaries vary depending on architecture, compilation toolchain settings, compilation flags, link options, and all sorts of other things. There's really no way to address those built artifacts even if they were available on pypi or something.
- Filligree 4y ago
  Well, there is: It's Nix's buildPythonApplication. Trying to use Python in Nix is frequently a bother, but at least once it works, it works.
- structural 4y ago
  There's a couple different ways. 1. Build the artifact that you depend on, publish it on "pypi or something" referenced by both the version of the library and a hash that uniquely identifies the build. This is the approach taken by conda, for better or worse. 2. Only allow one canonical build of a library you depend on so that the "version" becomes a unique identifier for the build. This is the approach taken by distribution package managers. 3. Create metapackages that describe ABI constraints that are required by packages and must be satisfied by the underlying system. For example, a "cxxabi" package could be provided by the underlying system, and the packaging tools could automatically add dependencies on "cxxabi" at build time, based on either an exact pin to the library built against, or in some cases, a relaxed dependency by inspection of versioned symbols used by the binary. 4. Statically link all your dependencies and/or vendor all your dependencies. These are used by quite a few pypi packages that depend on standalone C libraries to avoid most of the issues altogether.