7 ms·

ABI compatibility in Python: How hard could it be?

96 points by lukastyrychtr 4y ago

frozenport 4y ago
One reading is that despite declared mismatches between versions. There were no actual instances of incompatibility found?
- SAI_Peregrinus 4y ago
  Using symbols from a newer Python version with an older ohe declared is an incompatibility, if run on an older Python install.
avidphantasm 4y ago
May take on this, as a Python specialist who maintains a C++ library that exposes Python bindings (but not an expert in such), is that the Python packaging ecosystem (as it has organically grown) is a mess. I would like an authoritative (i.e., in the standard library) way to accomplish the major packaging tasks. I shouldn’t have to track down the current canonical way to do things from two or three sources.
- deleted 4y ago
  [deleted]
- noisenotsignal 4y ago
  Agree. I had a build pipeline pulling down an internal package to build into a `.pex` [0] file for running a Python program in a release pipeline that did not have access to the internal package repository. I came up with this solution because the typical virtualenv that you might build locally does not appreciate being copied wholesale to another system (iirc it had some absolute paths). One day the release pipeline broke because a dependency's authors published a new version containing a wheel for a newer ABI, though the older ABI was still supported. My build pipeline pulled the one for the later ABI because it was compatible, but the release pipeline environment, which could not be upgraded, did not have the newer version of gcc required. It was a nasty introduction to the fact that one Python package version can have multiple published wheels targeting different C++ compilers! But the worst part was not that I learned something new! It was that I wrangled with both pex and pip for a few hours trying to figure out how to download a wheel for an older ABI as both tools resolve the latest compatible by default. The options to do so are ostensibly there but the effect was not. I definitely could have made some mistakes but in my mind it shouldn't be _hard_ to get it right. I ended up downloading the wheel for the internal package in the build and copying it directly into the artifact. The release pipeline resolved public dependencies and installed the internal package from the local file. [0] - https://pex.readthedocs.io/en/latest/ https://pex.readthedocs.io/en/latest/
  - orf 4y ago
    The release pipeline broke because you didn't lock your dependencies.
    - noisenotsignal 4y ago
      Not sure what you mean. pex is the dependency lock, at least from a version perspective. To be clear, it broke after I intentionally upgraded the offending package to the later version. I admit to being unaware of "versions inside versions" where a version may have multiple published ABIs that are not compatible across systems, but a nice packaging system would still make it easy for me to use platform X to build for platform Y.
      - orf 4y ago
        Oh, from your "One day the release pipeline broke because a dependency's authors published a new version..." wording it seemed that this breakage occurred without you upgrading anything.
        noisenotsignal 4y ago
        Sorry for the wording, I see how it can be interpreted that way. Seems I skipped a few details on accident.
  - bsder 4y ago
    > It was a nasty introduction to the fact that one Python package version can have multiple published wheels targeting different C++ compilers! Erm, is this not the failure of C++ standards instead of Python? This seems like Python is the failure point being caused by C++ not having a stable ABI. Python is simply trying to paper over the brokenness of the C++ ecosystem. Am I missing something?
    - noisenotsignal 4y ago
      That’s a good point! However, there’s still things Python could do to help make the experience better. The package I was using had multiple artifacts per version, including the raw zipped source code and wheels for each ABI they built for. It’s certainly convenient that the most applicable one for your current system is pulled automatically by pip. But in this case that’s not the behavior I wanted, and I could not successfully get pex or pip to download a “less optimal” artifact. You can blame C++, but if Python is already papering over deficiencies it’s not unreasonable to expect improvements that aren’t fundamentally impossible. It should definitely be possible for there to be one easily understood argument to specify which artifact pip should download. In my (potentially flawed) experience, figuring out which ABI value to pass was already not straightforward. Even then, the arguments for selecting an ABI or even asking for the raw source instead of a wheel weren’t seeming to have an effect.
- inferiorhuman 4y ago
  It always struck me as a bit ironic that such an opinionated language could utterly fail to have "one right way" to package things. As much as Rust's aversion to binaries can be a pain, building and packaging (even cross building) is generally painless.
  - 323 4y ago
    That's because Rust is tiny and new. Wait until you have Rust packages which include and wrap hundred of thousands of lines of very old C and Fortran code like numpy has.
    - inferiorhuman 4y ago
      Nah. It'd probably be surprising how many crates have dependencies here and there on C code – mostly it's seamless even when cross compiling. I've gotten (ages ago) a cross toolchain setup to build binaries for one of those MIPS routers and the biggest pain point was being stuck on an antiquated Debian where cross compiling was painful. There are popularity issues with Rust (e.g. crate namespacing), but non-Rust code hasn't been one of them for me. So far Conda is pretty much the only problem I've had with native code mingling with Python (although pip was able to get numpy installed on my BSD box). Otherwise it's an issue of the tooling just being really archaic. Which is amusing to me given how much hand wringing there is over providing one right, idiomatic way to do things (e.g. ternary operator). For me digging into Stable Diffusion (and related bits and bobs) was a bit of an eye opener with just how much cargo culting is going on. Installation instructions are often copy and pasted from other related projects. Or maybe just check out our repo and follow the instructions from some other project. Honestly, my hope is that some of the effort spent obsessing about language details gets spent on coming up with a better packaging and dependency management story.
    - fiedzia 4y ago
      > That's because Rust is tiny and new. Dozens of other languages that are tiny and new do not have good packaging story. Rust does, because it cares about developer experience. > Wait until you have Rust packages which include and wrap hundred of thousands of lines of very old C and Fortran code like numpy has. I don't know about Fortran, but C is basically legacy at this point, it doesn't change much and even if it does, there is one some-c-lib wrapper package that takes care of all those issues, Rust apps rarely depend on some C dependency directly.
- blooalien 4y ago
  https://www.pypa.io/ https://www.pypa.io/ ? "The PyPA publishes the Python Packaging User Guide [1], which is the authoritative resource on how to package, publish, and install Python projects using current tools." [1]: https://packaging.python.org/ https://packaging.python.org/
  - avidphantasm 4y ago
    Last I checked, and I could be missing something, PyPA doesn’t say how to package extensions using the new pyproject.toml style. I happen to be using SWIG (which I inherited), which suggests the use of setup.py alone. I just haven’t had time to figure out the right path.
    - blooalien 4y ago
      I dunno about using it with SWIG but I've had a pretty good experience using Poetry [1] for pure Python stuff myself. That uses pyproject.toml to declare all your project's dependencies, but no idea if it can be used with SWIG or not. I'd have to research it… [1]: https://python-poetry.org/ https://python-poetry.org/ Edit: Some quick cursory research seems to indicate I'd have to do some custom scripting to integrate SWIG into the process.
      - filmor 4y ago
        Poetry is currently not useful for extensions: https://github.com/python-poetry/poetry/issues/2740 https://github.com/python-poetry/poetry/issues/2740
    - filmor 4y ago
      You can migrate a lot out of your setup.py into the pyproject.toml nowadays (https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html https://setuptools.pypa.io/en/latest/userguide/pyproject_con...).
    - xapata 4y ago
      Wouldn't you make a wheel?
beagle3 4y ago
Related: Nimpy[0] provides an easy way to write Python extensions in Nim, which manages the ABI side very well. Python 2 is now gone, but until it was, Nimpy was an easy way to write Python extension modules that only needed to be compiled once, and would work with any of your installed Python 2 and Python 3. Magic. [0] https://github.com/yglukhov/nimpy https://github.com/yglukhov/nimpy
somat 4y ago
I don't like it. once people have abi guarantees they start thinking they can skip shipping the source.
- ShowalkKama 4y ago
  No worries, some people already do! I've seen plenty of packages with no source release.
  - pabs3 4y ago
    I wish PyPI required reproducible builds, including public source code. https://reproducible-builds.org/ https://reproducible-builds.org/
jnwatson 4y ago
I maintain a small open source Python module written in C. I can't use the limited API (I use stuff outside it), so I have to maintain a build matrix. If you want to see what a decent-sized Python build matrix looks like, take a gander: https://github.com/jnwatson/py-lmdb/actions/runs/3457851983 https://github.com/jnwatson/py-lmdb/actions/runs/3457851983 It gets out of hand pretty fast.
- ramesh31 4y ago
  Have you considered WASM?
  - int_19h 4y ago
    Python on wasm is still very much a work-in-progress. But even if it weren't, wasm wheels are only compatible with wasm Python, which is not what most users of your library out there are going to have.
  - jnwatson 4y ago
    PRs accepted.
- nshm 4y ago
  And you didn't even include crazy people using Alpine Linux with musl.
nshm 4y ago
After many years of Python ABI pain we moved to python-cffi. Life become way easier.
MiguelX413 4y ago
I'm so glad I don't have to worry about these things as a PyO3 user
ngcc_hk 4y ago
ABI discussion is one thing. But if you into the discussion there is another - how to freeze the dependence … the more important question is transparency. It is like docker sort of where you build up your final based on certain before image (binary c is the problem mentioned most). Can we have a way to document it all or even optional download all for the production software then freeze it. And testing document in a .sw_env_conf to keep the change vs the base. If it is a re-runnable and readable doc we can trace it. Sorry not solving the Abi but try to deal with the mess. Python is easy to learn and program. But running it is really. I document at least 7 Python version and control which one to use is itself hard.

96 points by lukastyrychtr 4y ago

One reading is that despite declared mismatches between versions. There were no actual instances of incompatibility found?

SAI_Peregrinus 4y ago
Using symbols from a newer Python version with an older ohe declared is an incompatibility, if run on an older Python install.

avidphantasm 4y ago

May take on this, as a Python specialist who maintains a C++ library that exposes Python bindings (but not an expert in such), is that the Python packaging ecosystem (as it has organically grown) is a mess. I would like an authoritative (i.e., in the standard library) way to accomplish the major packaging tasks. I shouldn’t have to track down the current canonical way to do things from two or three sources.

deleted 4y ago
[deleted]
noisenotsignal 4y ago
Agree. I had a build pipeline pulling down an internal package to build into a `.pex` [0] file for running a Python program in a release pipeline that did not have access to the internal package repository. I came up with this solution because the typical virtualenv that you might build locally does not appreciate being copied wholesale to another system (iirc it had some absolute paths). One day the release pipeline broke because a dependency's authors published a new version containing a wheel for a newer ABI, though the older ABI was still supported. My build pipeline pulled the one for the later ABI because it was compatible, but the release pipeline environment, which could not be upgraded, did not have the newer version of gcc required. It was a nasty introduction to the fact that one Python package version can have multiple published wheels targeting different C++ compilers! But the worst part was not that I learned something new! It was that I wrangled with both pex and pip for a few hours trying to figure out how to download a wheel for an older ABI as both tools resolve the latest compatible by default. The options to do so are ostensibly there but the effect was not. I definitely could have made some mistakes but in my mind it shouldn't be _hard_ to get it right. I ended up downloading the wheel for the internal package in the build and copying it directly into the artifact. The release pipeline resolved public dependencies and installed the internal package from the local file. [0] - https://pex.readthedocs.io/en/latest/ https://pex.readthedocs.io/en/latest/
- orf 4y ago
  The release pipeline broke because you didn't lock your dependencies.
  - noisenotsignal 4y ago
    Not sure what you mean. pex is the dependency lock, at least from a version perspective. To be clear, it broke after I intentionally upgraded the offending package to the later version. I admit to being unaware of "versions inside versions" where a version may have multiple published ABIs that are not compatible across systems, but a nice packaging system would still make it easy for me to use platform X to build for platform Y.
    - orf 4y ago
      Oh, from your "One day the release pipeline broke because a dependency's authors published a new version..." wording it seemed that this breakage occurred without you upgrading anything.
      - noisenotsignal 4y ago
        Sorry for the wording, I see how it can be interpreted that way. Seems I skipped a few details on accident.
- bsder 4y ago
  > It was a nasty introduction to the fact that one Python package version can have multiple published wheels targeting different C++ compilers! Erm, is this not the failure of C++ standards instead of Python? This seems like Python is the failure point being caused by C++ not having a stable ABI. Python is simply trying to paper over the brokenness of the C++ ecosystem. Am I missing something?
  - noisenotsignal 4y ago
    That’s a good point! However, there’s still things Python could do to help make the experience better. The package I was using had multiple artifacts per version, including the raw zipped source code and wheels for each ABI they built for. It’s certainly convenient that the most applicable one for your current system is pulled automatically by pip. But in this case that’s not the behavior I wanted, and I could not successfully get pex or pip to download a “less optimal” artifact. You can blame C++, but if Python is already papering over deficiencies it’s not unreasonable to expect improvements that aren’t fundamentally impossible. It should definitely be possible for there to be one easily understood argument to specify which artifact pip should download. In my (potentially flawed) experience, figuring out which ABI value to pass was already not straightforward. Even then, the arguments for selecting an ABI or even asking for the raw source instead of a wheel weren’t seeming to have an effect.
inferiorhuman 4y ago
It always struck me as a bit ironic that such an opinionated language could utterly fail to have "one right way" to package things. As much as Rust's aversion to binaries can be a pain, building and packaging (even cross building) is generally painless.
- 323 4y ago
  That's because Rust is tiny and new. Wait until you have Rust packages which include and wrap hundred of thousands of lines of very old C and Fortran code like numpy has.
  - inferiorhuman 4y ago
    Nah. It'd probably be surprising how many crates have dependencies here and there on C code – mostly it's seamless even when cross compiling. I've gotten (ages ago) a cross toolchain setup to build binaries for one of those MIPS routers and the biggest pain point was being stuck on an antiquated Debian where cross compiling was painful. There are popularity issues with Rust (e.g. crate namespacing), but non-Rust code hasn't been one of them for me. So far Conda is pretty much the only problem I've had with native code mingling with Python (although pip was able to get numpy installed on my BSD box). Otherwise it's an issue of the tooling just being really archaic. Which is amusing to me given how much hand wringing there is over providing one right, idiomatic way to do things (e.g. ternary operator). For me digging into Stable Diffusion (and related bits and bobs) was a bit of an eye opener with just how much cargo culting is going on. Installation instructions are often copy and pasted from other related projects. Or maybe just check out our repo and follow the instructions from some other project. Honestly, my hope is that some of the effort spent obsessing about language details gets spent on coming up with a better packaging and dependency management story.
  - fiedzia 4y ago
    > That's because Rust is tiny and new. Dozens of other languages that are tiny and new do not have good packaging story. Rust does, because it cares about developer experience. > Wait until you have Rust packages which include and wrap hundred of thousands of lines of very old C and Fortran code like numpy has. I don't know about Fortran, but C is basically legacy at this point, it doesn't change much and even if it does, there is one some-c-lib wrapper package that takes care of all those issues, Rust apps rarely depend on some C dependency directly.
blooalien 4y ago
https://www.pypa.io/ https://www.pypa.io/ ? "The PyPA publishes the Python Packaging User Guide [1], which is the authoritative resource on how to package, publish, and install Python projects using current tools." [1]: https://packaging.python.org/ https://packaging.python.org/
- avidphantasm 4y ago
  Last I checked, and I could be missing something, PyPA doesn’t say how to package extensions using the new pyproject.toml style. I happen to be using SWIG (which I inherited), which suggests the use of setup.py alone. I just haven’t had time to figure out the right path.
  - blooalien 4y ago
    I dunno about using it with SWIG but I've had a pretty good experience using Poetry [1] for pure Python stuff myself. That uses pyproject.toml to declare all your project's dependencies, but no idea if it can be used with SWIG or not. I'd have to research it… [1]: https://python-poetry.org/ https://python-poetry.org/ Edit: Some quick cursory research seems to indicate I'd have to do some custom scripting to integrate SWIG into the process.
    - filmor 4y ago
      Poetry is currently not useful for extensions: https://github.com/python-poetry/poetry/issues/2740 https://github.com/python-poetry/poetry/issues/2740
  - filmor 4y ago
    You can migrate a lot out of your setup.py into the pyproject.toml nowadays (https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html https://setuptools.pypa.io/en/latest/userguide/pyproject_con...).
  - xapata 4y ago
    Wouldn't you make a wheel?

beagle3 4y ago

Related: Nimpy[0] provides an easy way to write Python extensions in Nim, which manages the ABI side very well. Python 2 is now gone, but until it was, Nimpy was an easy way to write Python extension modules that only needed to be compiled once, and would work with any of your installed Python 2 and Python 3. Magic. [0] https://github.com/yglukhov/nimpy https://github.com/yglukhov/nimpy

somat 4y ago

I don't like it. once people have abi guarantees they start thinking they can skip shipping the source.

ShowalkKama 4y ago
No worries, some people already do! I've seen plenty of packages with no source release.
- pabs3 4y ago
  I wish PyPI required reproducible builds, including public source code. https://reproducible-builds.org/ https://reproducible-builds.org/

jnwatson 4y ago

I maintain a small open source Python module written in C. I can't use the limited API (I use stuff outside it), so I have to maintain a build matrix. If you want to see what a decent-sized Python build matrix looks like, take a gander: https://github.com/jnwatson/py-lmdb/actions/runs/3457851983 https://github.com/jnwatson/py-lmdb/actions/runs/3457851983 It gets out of hand pretty fast.

ramesh31 4y ago
Have you considered WASM?
- int_19h 4y ago
  Python on wasm is still very much a work-in-progress. But even if it weren't, wasm wheels are only compatible with wasm Python, which is not what most users of your library out there are going to have.
- jnwatson 4y ago
  PRs accepted.
nshm 4y ago
And you didn't even include crazy people using Alpine Linux with musl.

nshm 4y ago

After many years of Python ABI pain we moved to python-cffi. Life become way easier.

MiguelX413 4y ago

I'm so glad I don't have to worry about these things as a PyO3 user

ngcc_hk 4y ago

ABI discussion is one thing. But if you into the discussion there is another - how to freeze the dependence … the more important question is transparency. It is like docker sort of where you build up your final based on certain before image (binary c is the problem mentioned most). Can we have a way to document it all or even optional download all for the production software then freeze it. And testing document in a .sw_env_conf to keep the change vs the base. If it is a re-runnable and readable doc we can trace it. Sorry not solving the Abi but try to deal with the mess. Python is easy to learn and program. But running it is really. I document at least 7 Python version and control which one to use is itself hard.