4 ms·

What you’re proposing here is essentially “if we fix the C/C++ build systems environment, this would be easy!”. You’re absolutely right, but fixing that mess ha

What you’re proposing here is essentially “if we fix the C/C++ build systems environment, this would be easy!”. You’re absolutely right, but fixing that mess has been a multi-decade goal that’s gone nowhere. One of the great victories of new systems languages like Rust and Zig is that they standardized build systems. But untangling each individual dependency’s pile of terrible CMake (or autoconf, or vcxproj) hacks is a project in itself, and it’s often a deeply political one tied up with the unique history of each project.

HelloNurse 2y ago
The typical build scripts and tools and terrible hacks that are adequate for a standalone C/C++/Fortran project are lacking for building the same project as a Python extension, which requires portability and supported, not too custom, build steps. The ambition and usefulness of aiming for the latter higher standard is quite new: it has gone (relatively) nowhere because it has been a multi-decade non-goal, which only a small minority of users cares about.
- theamk 2y ago
  Hard-to-build python extensions are basically .so files with some special symbols exposed. As others said, a general task to "build libfoo.so" is very complex, and currently requires a wide variety of build systems. I don't see why this will get any easier if we require this .so file to export some Python-specific symbols.
kristoff_it 2y ago
> What you’re proposing here is essentially “if we fix the C/C++ build systems environment, this would be easy!”. You’re absolutely right, but fixing that mess has been a multi-decade goal that’s gone nowhere. Not sure I would call it easy, as it would still take a lot of effort to update how PyPI works to account for these new capabilities, but that's exactly what the Zig compiler & build system solved. Rust is completely hands off when it comes to C/C++ dependencies, Zig can package and build them. That's why I created https://github.com/allyourcodebase/ https://github.com/allyourcodebase/. As I've mentioned on lobsters, look a this example `build.zig.zon` file: https://github.com/allyourcodebase/srt/blob/main/build.zig.zon https://github.com/allyourcodebase/srt/blob/main/build.zig.z... It mentions 3 dependencies: Haivision/srt, the upstream C++ project mbedtls googletest When you run zig build all these 3 dependencies are downloaded and their build.zig is run if present (the first one doesn’t have a build.zig since it’s just the vanilla C/C++ upstream project that we are providing a build script for). The work to package everything must still happen, but once it’s done correctly you get the ability to build from any host for any target, which you can literally see happening in the CI runs of that repo: https://github.com/allyourcodebase/srt/actions/runs/10982569887 https://github.com/allyourcodebase/srt/actions/runs/10982569... This kind of skepticism is exactly why I wrote this post. The details of actually coming up with a realistic upgrade path for PyPI are certainly much more nuanced than what I wrote in the post, but the core insight is that the C/C++ build intractability problem has been solved... and that you shouldn't depend on free big tech money if you can.
- ddulaney 2y ago
  In my experience, `zig cc` is great at cross-compiling (which is a hard problem that it solves!) but doesn't actually help with the rest of the problem. You still have to run cmake/autoconf/meson/whatever else, which is the part that's project-specific and often quite fiddly.
  - lifthrasiir 2y ago
    This has been my experience as well. I previously used cargo-zigbuild for packaging at work (and contributed both to cargo-zigbuild and Zig as a byproduct), and I still had several troubles that I had to analyze and tackle in spite of them.
  - kristoff_it 2y ago
    Well if you're limiting yourself to zig cc, then all you get is a C compiler. If you take the time to kick out of your project that soup of build systems and replace them with a build.zig, then you got yourself a complete solution. Takes some effort but it's perfectly doable, see https://github.com/allyourcodebase https://github.com/allyourcodebase
- HelloNurse 2y ago
  The "upgrade path" is for individual packages (providing portable build scripts) and for client-side Python package management (actually providing and running next-generation tools), not for PyPI which already supports package metadata stating what platforms a package is compatible for.
sitkack 2y ago
They do, Zig ships clang. The easiest way to install clang is via pip. pip install ziglang python -m ziglang clang --version clang version 18.1.6 (https://github.com/ziglang/zig-bootstrap 98bc6bf4fc4009888d33941daf6b600d20a42a56) Target: aarch64-unknown-darwin23.6.0 Thread model: posix InstalledDir: /usr/bin Very soon, Zig will all your (code)base.
- kbolino 2y ago
  Zig is divorcing itself from Clang: https://github.com/ziglang/zig/issues/16270 https://github.com/ziglang/zig/issues/16270
  - sitkack 2y ago
    Zig is and will be shipping clang for a great long while. Not only will it be shipping clang so you can build other code with it, they aren't removing LLVM support. https://github.com/ziglang/zig/issues/16270#issuecomment-2058934250 https://github.com/ziglang/zig/issues/16270#issuecomment-205...
    - kbolino 2y ago
      I raise you https://github.com/ziglang/zig/issues/20875 https://github.com/ziglang/zig/issues/20875 It is not entirely clear what any of this means for the future, and in any case, it keeps changing all the time. The promise to maintain LLVM support just applies to Zig code. That doesn't require Clang. LLVM alone is not a C compiler, and neither is Zig without Clang. What very well may happen is that the "standard" distribution of Zig will continue to bundle Clang, while a "minimal" distribution also gets created which omits Clang, and thus cannot compile C code. However, none of this has been spelled out clearly yet, and so I think it's reasonable to say don't depend on Zig to give you Clang.
      - sitkack 2y ago
        Zig is migrating away from LLVM as the mandatory backend, it will still ship clang and llvm so that it can build everything it needs which is critical for its mission for all your (code) base.

HelloNurse 2y ago

The typical build scripts and tools and terrible hacks that are adequate for a standalone C/C++/Fortran project are lacking for building the same project as a Python extension, which requires portability and supported, not too custom, build steps. The ambition and usefulness of aiming for the latter higher standard is quite new: it has gone (relatively) nowhere because it has been a multi-decade non-goal, which only a small minority of users cares about.

theamk 2y ago
Hard-to-build python extensions are basically .so files with some special symbols exposed. As others said, a general task to "build libfoo.so" is very complex, and currently requires a wide variety of build systems. I don't see why this will get any easier if we require this .so file to export some Python-specific symbols.

kristoff_it 2y ago

> What you’re proposing here is essentially “if we fix the C/C++ build systems environment, this would be easy!”. You’re absolutely right, but fixing that mess has been a multi-decade goal that’s gone nowhere. Not sure I would call it easy, as it would still take a lot of effort to update how PyPI works to account for these new capabilities, but that's exactly what the Zig compiler & build system solved. Rust is completely hands off when it comes to C/C++ dependencies, Zig can package and build them. That's why I created https://github.com/allyourcodebase/ https://github.com/allyourcodebase/. As I've mentioned on lobsters, look a this example `build.zig.zon` file: https://github.com/allyourcodebase/srt/blob/main/build.zig.zon https://github.com/allyourcodebase/srt/blob/main/build.zig.z... It mentions 3 dependencies: Haivision/srt, the upstream C++ project mbedtls googletest When you run zig build all these 3 dependencies are downloaded and their build.zig is run if present (the first one doesn’t have a build.zig since it’s just the vanilla C/C++ upstream project that we are providing a build script for). The work to package everything must still happen, but once it’s done correctly you get the ability to build from any host for any target, which you can literally see happening in the CI runs of that repo: https://github.com/allyourcodebase/srt/actions/runs/10982569887 https://github.com/allyourcodebase/srt/actions/runs/10982569... This kind of skepticism is exactly why I wrote this post. The details of actually coming up with a realistic upgrade path for PyPI are certainly much more nuanced than what I wrote in the post, but the core insight is that the C/C++ build intractability problem has been solved... and that you shouldn't depend on free big tech money if you can.

ddulaney 2y ago
In my experience, `zig cc` is great at cross-compiling (which is a hard problem that it solves!) but doesn't actually help with the rest of the problem. You still have to run cmake/autoconf/meson/whatever else, which is the part that's project-specific and often quite fiddly.
- lifthrasiir 2y ago
  This has been my experience as well. I previously used cargo-zigbuild for packaging at work (and contributed both to cargo-zigbuild and Zig as a byproduct), and I still had several troubles that I had to analyze and tackle in spite of them.
- kristoff_it 2y ago
  Well if you're limiting yourself to zig cc, then all you get is a C compiler. If you take the time to kick out of your project that soup of build systems and replace them with a build.zig, then you got yourself a complete solution. Takes some effort but it's perfectly doable, see https://github.com/allyourcodebase https://github.com/allyourcodebase
HelloNurse 2y ago
The "upgrade path" is for individual packages (providing portable build scripts) and for client-side Python package management (actually providing and running next-generation tools), not for PyPI which already supports package metadata stating what platforms a package is compatible for.

sitkack 2y ago

They do, Zig ships clang. The easiest way to install clang is via pip. pip install ziglang python -m ziglang clang --version clang version 18.1.6 (https://github.com/ziglang/zig-bootstrap 98bc6bf4fc4009888d33941daf6b600d20a42a56) Target: aarch64-unknown-darwin23.6.0 Thread model: posix InstalledDir: /usr/bin Very soon, Zig will all your (code)base.

kbolino 2y ago
Zig is divorcing itself from Clang: https://github.com/ziglang/zig/issues/16270 https://github.com/ziglang/zig/issues/16270
- sitkack 2y ago
  Zig is and will be shipping clang for a great long while. Not only will it be shipping clang so you can build other code with it, they aren't removing LLVM support. https://github.com/ziglang/zig/issues/16270#issuecomment-2058934250 https://github.com/ziglang/zig/issues/16270#issuecomment-205...
  - kbolino 2y ago
    I raise you https://github.com/ziglang/zig/issues/20875 https://github.com/ziglang/zig/issues/20875 It is not entirely clear what any of this means for the future, and in any case, it keeps changing all the time. The promise to maintain LLVM support just applies to Zig code. That doesn't require Clang. LLVM alone is not a C compiler, and neither is Zig without Clang. What very well may happen is that the "standard" distribution of Zig will continue to bundle Clang, while a "minimal" distribution also gets created which omits Clang, and thus cannot compile C code. However, none of this has been spelled out clearly yet, and so I think it's reasonable to say don't depend on Zig to give you Clang.
    - sitkack 2y ago
      Zig is migrating away from LLVM as the mandatory backend, it will still ship clang and llvm so that it can build everything it needs which is critical for its mission for all your (code) base.