4 ms·

Is my understanding correct that this would provide version agnostic python bindings? Currently, I am building a version of my bindings separately for each vers

by koe123 2y ago

Is my understanding correct that this would provide version agnostic python bindings? Currently, I am building a version of my bindings separately for each version (e.g. building and linking with python 3.7, 3.8, etc.). While automated, it still makes CI/CD take quite a long time.

gjvc 2y ago
While automated, it still makes CI/CD take quite a long time See about using ccache -- https://ccache.dev/ https://ccache.dev/
- IshKebab 2y ago
  I wouldn't recommend ccache (or sccache) in CI unless you really need it. They are not 100% reliable, and any time you save from caching will be more than lost debugging the weird failures you get when they go wrong.
  - gjvc 2y ago
    please provide evidence for this assertion.
    - IshKebab 2y ago
      Why are you so skeptical? Think about how it works and then you'll understand that cache invalidation bugs are completely inevitable. Hell, cache invalidation is notoriously difficult to get right even when you aren't building it on top of a complex tool that was never designed for aggressive caching. Just search the bugs for "hash": https://github.com/ccache/ccache/issues?q=is%3Aissue+hash+is%3Aclosed+label%3Abug https://github.com/ccache/ccache/issues?q=is%3Aissue+hash+is...
    - imtringued 2y ago
      You can't cache based on the file contents alone. You will also need to cache based on all OS/compiler queries/variables/settings that the preprocessor depends on, since the header files might generate completely different content based on what ifdef gets triggered.
      - mananaysiempre 2y ago
        And that’s not impossible, just tedious. One tricky (and often unimportant) part is negative dependencies—when the build depends on the fact that a header or library cannot be found in a particular directory on a search path (which happens all the time, if you think about it). As far as I know, no compilers will cooperate with you on this, so build systems that try to get this right have to trace the compiler’s system calls to be sure (Tup does something like this) or completely control and hash absolutely everything that the compiler could possibly see (Nix and IIUC Bazel).
        zorgmonkey 2y ago
        In C++ the __has_include preprocessor expression has been standardized since C++17, I'm not certain if C has standardized it yet though.
        mananaysiempre 2y ago
        It’s not about that, that’s not relevant to ccache at all. (And yes, C23 does have __has_include, though not a lot of compilers have C23 yet.) It’s about having potentially conflicting headers in the source file’s directory, in your -I directories, and in your /usr/include directories. Suppose a previous compile correctly resolved <libfoo.h> to /usr/include/libfoo.h, and that file remains unchanged, but since that time you’ve installed a private build of libfoo such that a new compile would instead resolve that to ~/.local/include/libfoo.h. What you want is to record not just that your compile opened /usr/include/libfoo.h (“positive dependencies” you get with -MD et al.), but that it tried $GITHOME/include/libfoo.h, ~/.local/include/libfoo.h, etc. before that and failed (“negative dependencies”), so that if any of those appear later you can force a recompile.
        zorgmonkey 2y ago
        Oh yeah that can cause lots of weird problems. I've run into that sort of issue a lot when cross-compiling, cause often then you might have a system copy of a library and a different version for the target, that can be a real pain.
    - cstrahan 2y ago
      Here’s an example of a common Rust dependency which had a very subtle caching issue: https://github.com/dtolnay/thiserror/pull/279 https://github.com/dtolnay/thiserror/pull/279
aragilar 2y ago
I believe so, but it would presumably depend on what features you use.
kzrdude 2y ago
Cpython also has a limited stable abi and cp3X-abi3 wheels are compatible across multiple versions of Python. https://docs.python.org/3/c-api/stable.html https://docs.python.org/3/c-api/stable.html
- deleted 2y ago
  [deleted]
- mardifoufs 2y ago
  But it is very limited. Understandably so, as they don't want to ossify the internal APIs, but it still is so limited that you can't actually build anything just using just that API as far as I know.
  - kzrdude 2y ago
    I checked now - polars is built using py abi3 wheels, and for me that means that you can build something substantial using the stable ABI! :) See https://pypi.org/project/polars/1.9.0/#files https://pypi.org/project/polars/1.9.0/#files cp38-abi3 wheels means they are compatible with cpython 3.8 or later.
    - masklinn 2y ago
      That makes sense: I would assume polars mostly converts from Python to rust at the edge then it works in rust internally. Though I’ve not really looked at the details I’d assume most of the missing stuff would be “intimate” APIs of builtin types. And all the macros leveraging implementation details.
    - mardifoufs 2y ago
      Woah! Okay, that's very cool. I thought it was much more limited than that (for the stable abi). Awesome! It seems like they use mostly normal python as a bridge with the rust codebase. So from what I've seen on their repo, they mostly do not use any CPython APIs (a part from a few wrappers I think). Which makes sense!
masklinn 2y ago
You can already build a single wheel as long as you only target cpython, if your needs fit with the limited / stable abi (abi3). While pypy and graal have API support they don't have abi / abi3 support, so they still have to be built on their own (and per version I think).
filmor 2y ago
As others have said, this has been supported since the limited/stable APIs were introduced. What this adds is a way of implementing a Python extension that can be loaded in (not just compiled for, which is already an improvement!) different Python implementations, namely CPython, Pypy and GraalVM.

by koe123 2y ago

gjvc 2y ago

While automated, it still makes CI/CD take quite a long time See about using ccache -- https://ccache.dev/ https://ccache.dev/

IshKebab 2y ago
I wouldn't recommend ccache (or sccache) in CI unless you really need it. They are not 100% reliable, and any time you save from caching will be more than lost debugging the weird failures you get when they go wrong.
- gjvc 2y ago
  please provide evidence for this assertion.
  - IshKebab 2y ago
    Why are you so skeptical? Think about how it works and then you'll understand that cache invalidation bugs are completely inevitable. Hell, cache invalidation is notoriously difficult to get right even when you aren't building it on top of a complex tool that was never designed for aggressive caching. Just search the bugs for "hash": https://github.com/ccache/ccache/issues?q=is%3Aissue+hash+is%3Aclosed+label%3Abug https://github.com/ccache/ccache/issues?q=is%3Aissue+hash+is...
  - imtringued 2y ago
    You can't cache based on the file contents alone. You will also need to cache based on all OS/compiler queries/variables/settings that the preprocessor depends on, since the header files might generate completely different content based on what ifdef gets triggered.
    - mananaysiempre 2y ago
      And that’s not impossible, just tedious. One tricky (and often unimportant) part is negative dependencies—when the build depends on the fact that a header or library cannot be found in a particular directory on a search path (which happens all the time, if you think about it). As far as I know, no compilers will cooperate with you on this, so build systems that try to get this right have to trace the compiler’s system calls to be sure (Tup does something like this) or completely control and hash absolutely everything that the compiler could possibly see (Nix and IIUC Bazel).
      - zorgmonkey 2y ago
        In C++ the __has_include preprocessor expression has been standardized since C++17, I'm not certain if C has standardized it yet though.
        mananaysiempre 2y ago
        It’s not about that, that’s not relevant to ccache at all. (And yes, C23 does have __has_include, though not a lot of compilers have C23 yet.) It’s about having potentially conflicting headers in the source file’s directory, in your -I directories, and in your /usr/include directories. Suppose a previous compile correctly resolved <libfoo.h> to /usr/include/libfoo.h, and that file remains unchanged, but since that time you’ve installed a private build of libfoo such that a new compile would instead resolve that to ~/.local/include/libfoo.h. What you want is to record not just that your compile opened /usr/include/libfoo.h (“positive dependencies” you get with -MD et al.), but that it tried $GITHOME/include/libfoo.h, ~/.local/include/libfoo.h, etc. before that and failed (“negative dependencies”), so that if any of those appear later you can force a recompile.
        zorgmonkey 2y ago
        Oh yeah that can cause lots of weird problems. I've run into that sort of issue a lot when cross-compiling, cause often then you might have a system copy of a library and a different version for the target, that can be a real pain.
  - cstrahan 2y ago
    Here’s an example of a common Rust dependency which had a very subtle caching issue: https://github.com/dtolnay/thiserror/pull/279 https://github.com/dtolnay/thiserror/pull/279

aragilar 2y ago

I believe so, but it would presumably depend on what features you use.

kzrdude 2y ago

Cpython also has a limited stable abi and cp3X-abi3 wheels are compatible across multiple versions of Python. https://docs.python.org/3/c-api/stable.html https://docs.python.org/3/c-api/stable.html

deleted 2y ago
[deleted]
mardifoufs 2y ago
But it is very limited. Understandably so, as they don't want to ossify the internal APIs, but it still is so limited that you can't actually build anything just using just that API as far as I know.
- kzrdude 2y ago
  I checked now - polars is built using py abi3 wheels, and for me that means that you can build something substantial using the stable ABI! :) See https://pypi.org/project/polars/1.9.0/#files https://pypi.org/project/polars/1.9.0/#files cp38-abi3 wheels means they are compatible with cpython 3.8 or later.
  - masklinn 2y ago
    That makes sense: I would assume polars mostly converts from Python to rust at the edge then it works in rust internally. Though I’ve not really looked at the details I’d assume most of the missing stuff would be “intimate” APIs of builtin types. And all the macros leveraging implementation details.
  - mardifoufs 2y ago
    Woah! Okay, that's very cool. I thought it was much more limited than that (for the stable abi). Awesome! It seems like they use mostly normal python as a bridge with the rust codebase. So from what I've seen on their repo, they mostly do not use any CPython APIs (a part from a few wrappers I think). Which makes sense!

masklinn 2y ago

You can already build a single wheel as long as you only target cpython, if your needs fit with the limited / stable abi (abi3). While pypy and graal have API support they don't have abi / abi3 support, so they still have to be built on their own (and per version I think).

filmor 2y ago

As others have said, this has been supported since the limited/stable APIs were introduced. What this adds is a way of implementing a Python extension that can be loaded in (not just compiled for, which is already an improvement!) different Python implementations, namely CPython, Pypy and GraalVM.