5 ms·

Bazel and Glibc Versions

27 points by goranmoomin 1y ago

shykes 1y ago
It appears the Bazel community is getting closer to inventing containers.
- jeffbee 1y ago
  Bazel comes with ironclad sandboxing features, but people don't use them because build correctness is such a chore.
  - deleted 1y ago
    [deleted]
- paulddraper 1y ago
  Bazel has number of strategies, including containers. But that’s only a minority of what it does.
- klodolph 1y ago
  I don’t think there’s any real insights there. Containers tend to be coarse-grained. For example, maybe you are writing a program, so you put the entire build environment in a container. If you have lots of programs with different dependencies, do you need lots of containers? Do you need to rebuild lots of containers when dependencies change? Bazel is much more fine-grained, and in Bazel, each individual build action gets its own build environment. By default, that build environment includes the host (build system) glibc, compiler, system headers, system libraries, etc. If you want repeatable builds, you turn that off, and give Bazel the exact glibc, compiler, libraries, etc. to use for building your program. You get the isolation of containers, but much faster builds and better caching.
- josephg 1y ago
  You don’t need containers. Just hermetic builds. Ideally every byte read during the compilation process comes from a file under your control, that you version and you test. That includes the compiler, glibc, and all of your dependencies. Ambient, implicit dependencies are the devil’s playthings.
  - shykes 1y ago
    Right. But you need tooling to manage those dependencies, and an ecosystem of content to avoid reinventing the wheel. That's what containers provide. Since Bazel predates the modern container ecosystem, it had to invent its own tooling. The user experience is very rough, and the ecosystem tiny. As a result most Bazel users do not run their builds in a controlled environment - see the parent blog post. Adopting containers would solve this, but it seems to be a major blind spot for the Bazel community. My theory is that for them to adopt container technology, they will have to reinvent it themselves, hence my tongue-in-cheek comment.
fooblaster 1y ago
Why ain't the first answer.. use a bazel provided tool chain instead of using system toolchains? This article is totally mad.
- paulddraper 1y ago
  Agreed.
- jmmv 1y ago
  Because those Bazel toolchains don’t come with a glibc? How could they?
  - lokar 1y ago
    What? Of course they can, how is that hard?
    - jmmv 1y ago
      You need that glibc on the target system, and glibc doesn’t like static linking. How do you ship the built binary?
      - lokar 1y ago
        You include the target glibc as part of your toolchain And, I always prefer to actually pick my glibc independently from the host os and ship it with the binary (generally in an OCI image). This way you can patch/update the host without breaking the service.
        remexre 1y ago
        Does that result in working NSS?
        lokar 1y ago
        I normally statically link as much as possible and avoid nss, but you can make that work as well, just include it along with glibc.
notpushkin 1y ago
> the service crashes with a mysterious error: version 'GLIBC_2.28' not found Mysterious? This is by the way why many binary Python packages use https://github.com/pypa/manylinux https://github.com/pypa/manylinux for builds: if you build on an old glibc, your library will still (generally) work with newer versions.
10000truths 1y ago
One of the niceties of Zig's build system is its batteries-included support for targeting an explicitly provided glibc version. No need for alternate sysroots or dedicated old-distro containers like you would with the traditional C/C++ compilers, all you have to do is append it to the target triplet, like so: zig cc -target x86_64-linux-gnu.2.17 file.c
- klodolph 1y ago
  It’s not hard in Bazel either, it’s just not batteries-included.
klodolph 1y ago
JFC the author has done everything wrong here. A heterogenous build cluster with non-hermetic builds and shared caching. The fact that this is only a glibc symbol versioning problem and not something far more severe is, well, a blessing. At the bare fucking minimum, I would expect the CI builds to have a homogenous configuration and their own cache, not writable from outside CI. If you’re lazy, just wipe the cache every time you upgrade the CI cluster. Maybe I’ve just been living too long in environments where we care about artifact provenance and I’m blind to the whims of people who don’t care about provenance. I want to feel sympathetic, because I know Bazel is a pain in the ass to learn, but it sounds like the author was juggling knives in a windstorm and caught a few pointy ends.
r2vcap 1y ago
Whether you use Bazel or not, this is a well-known issue with an equally well-known solution. There’s no need for such a lengthy write-up: just use a consistent sysroot across your entire build environment. If you can’t create your own sysroot image, you can simply download Chromium’s prebuilt one and configure your C++ compile rules correctly. Problem solved.
- 01HNNWZ0MV43FF 1y ago
  Sounds like a jerb for Docker
- rockwotj 1y ago
  Yeah I built a custom sysroot for Redpanda (Bazel/C++/Distributed Kafka) using a really simple docker image: https://github.com/redpanda-data/redpanda/blob/dev/bazel/toolchain/Dockerfile.sysroot https://github.com/redpanda-data/redpanda/blob/dev/bazel/too... We also have an dockerfile for clang/LLVM in that repo so the whole thing is hermetic. It’s a bit of shame Bazel doesn’t come with stronger options/defaults here, because I feel like I want to reproduce this same toolchain on every C++ project with Bazel
  - DriftRegion 1y ago
    Thanks for sharing. As a non-google bazel user this is quite helpful.
cryptonector 1y ago
This happens in CIs. It's happened to me on GitHub Actions. The answer is always sysroots.
- Arrowmaster 1y ago
  I'm in the middle of submitting PRs to multiple projects because they are compiling on ubuntu-latest and forcing a glibc 2.38 requirement. These are multiplatform projects where most or none of the devs use Linux. The first project I was able to change their workflow to build inside a 20.04 container. The other project uses tauri and it requires some very recent libraries so I don't know if an older container will work. Do you have any documentation or generic recommendations for solving these issues caused by blindly using GitHub Actions for all compilations?
  - cryptonector 1y ago
    > The first project I was able to change their workflow to build inside a 20.04 container. This approach does _not_ work because you end up with the `node` that runs GitHub Actions not being able to run, certainly this will happen if you end using a sufficiently old container. > Do you have any documentation or generic recommendations for solving these issues caused by blindly using GitHub Actions for all compilations? Install these pkgs in an `ubuntu-latest` image: - debootstrap debian-archive-keyring - software-properties-common - schroot fakeroot fakechroot then - name: 'Cache sysroot' # This comes after checking out the sources because # actions/checkout@v4 cleans $PWD! id: cache-sysroot uses: actions/cache@v3 with: path: ${{ github.workspace }}/sysroot-DEBIAN_RELEASE key: sysroot-DEBIAN_RELEASE-${{ runner.os }}-${{ runner.arch }}-v1 - name: 'Setup cross-compilation sysroot' if: steps.cache-sysroot.oututs.cache-hit != 'true' run: | set -vx SYSROOT_PATH="${{ github.workspace }}/sysroot-DEBIAN_RELEASE" echo "SYSROOT_PATH=$SYSROOT_PATH" >> $GITHUB_ENV if [ ! -d sysroot-DEBIAN_RELEASE ]; then sudo debootstrap --arch=$(dpkg --print-architecture) DEBIAN_RELEASE sysroot-DEBIAN_RELEASE http://archive.ubuntu.com/ubuntu fi sudo chroot sysroot-DEBIAN_RELEASE apt-get update sudo chroot sysroot-DEBIAN_RELEASE apt-get install -y build-essential git wget curl sudo unzip zip autoconf libfreetype6-dev libcups2-dev libx11-dev libxext-dev libxrender-dev libxrandr-dev libxtst-dev libxt-dev libasound2-dev libffi-dev file binutils libfontconfig-dev sudo chroot sysroot-DEBIAN_RELEASE apt-get install -y software-properties-common sudo chroot sysroot-DEBIAN_RELEASE sudo add-apt-repository ppa:ubuntu-toolchain-r/test sudo chown -R $USER:$USER sysroot-DEBIAN_RELEASE where you replace `DEBIAN_RELEASE` with the release you want to target, and then - configure your project's build to use that sysroot. That's it. If your project does not support sysroots, make it do so. In general compilers will support sysroots, so it's just a matter of making your build configuration facility support sysroots.
setheron 1y ago
Am I the first to say "Nix solves this" Nix effectively ships a sysroot with every binary !
steeve 1y ago
This sandboxed LLVM toolchain completely solves that problem: https://github.com/cerisier/toolchains_llvm_bootstrapped https://github.com/cerisier/toolchains_llvm_bootstrapped
shaftway 1y ago
> The CI system picks it up, gets a cache hit from the developer’s build, and produces a release artifact. Why would you cache developer builds on CI?

27 points by goranmoomin 1y ago

shykes 1y ago

It appears the Bazel community is getting closer to inventing containers.

jeffbee 1y ago
Bazel comes with ironclad sandboxing features, but people don't use them because build correctness is such a chore.
- deleted 1y ago
  [deleted]
paulddraper 1y ago
Bazel has number of strategies, including containers. But that’s only a minority of what it does.
klodolph 1y ago
I don’t think there’s any real insights there. Containers tend to be coarse-grained. For example, maybe you are writing a program, so you put the entire build environment in a container. If you have lots of programs with different dependencies, do you need lots of containers? Do you need to rebuild lots of containers when dependencies change? Bazel is much more fine-grained, and in Bazel, each individual build action gets its own build environment. By default, that build environment includes the host (build system) glibc, compiler, system headers, system libraries, etc. If you want repeatable builds, you turn that off, and give Bazel the exact glibc, compiler, libraries, etc. to use for building your program. You get the isolation of containers, but much faster builds and better caching.
josephg 1y ago
You don’t need containers. Just hermetic builds. Ideally every byte read during the compilation process comes from a file under your control, that you version and you test. That includes the compiler, glibc, and all of your dependencies. Ambient, implicit dependencies are the devil’s playthings.
- shykes 1y ago
  Right. But you need tooling to manage those dependencies, and an ecosystem of content to avoid reinventing the wheel. That's what containers provide. Since Bazel predates the modern container ecosystem, it had to invent its own tooling. The user experience is very rough, and the ecosystem tiny. As a result most Bazel users do not run their builds in a controlled environment - see the parent blog post. Adopting containers would solve this, but it seems to be a major blind spot for the Bazel community. My theory is that for them to adopt container technology, they will have to reinvent it themselves, hence my tongue-in-cheek comment.

fooblaster 1y ago

Why ain't the first answer.. use a bazel provided tool chain instead of using system toolchains? This article is totally mad.

paulddraper 1y ago
Agreed.
jmmv 1y ago
Because those Bazel toolchains don’t come with a glibc? How could they?
- lokar 1y ago
  What? Of course they can, how is that hard?
  - jmmv 1y ago
    You need that glibc on the target system, and glibc doesn’t like static linking. How do you ship the built binary?
    - lokar 1y ago
      You include the target glibc as part of your toolchain And, I always prefer to actually pick my glibc independently from the host os and ship it with the binary (generally in an OCI image). This way you can patch/update the host without breaking the service.
      - remexre 1y ago
        Does that result in working NSS?
        lokar 1y ago
        I normally statically link as much as possible and avoid nss, but you can make that work as well, just include it along with glibc.

notpushkin 1y ago

> the service crashes with a mysterious error: version 'GLIBC_2.28' not found Mysterious? This is by the way why many binary Python packages use https://github.com/pypa/manylinux https://github.com/pypa/manylinux for builds: if you build on an old glibc, your library will still (generally) work with newer versions.

10000truths 1y ago

One of the niceties of Zig's build system is its batteries-included support for targeting an explicitly provided glibc version. No need for alternate sysroots or dedicated old-distro containers like you would with the traditional C/C++ compilers, all you have to do is append it to the target triplet, like so: zig cc -target x86_64-linux-gnu.2.17 file.c

klodolph 1y ago
It’s not hard in Bazel either, it’s just not batteries-included.

klodolph 1y ago

JFC the author has done everything wrong here. A heterogenous build cluster with non-hermetic builds and shared caching. The fact that this is only a glibc symbol versioning problem and not something far more severe is, well, a blessing. At the bare fucking minimum, I would expect the CI builds to have a homogenous configuration and their own cache, not writable from outside CI. If you’re lazy, just wipe the cache every time you upgrade the CI cluster. Maybe I’ve just been living too long in environments where we care about artifact provenance and I’m blind to the whims of people who don’t care about provenance. I want to feel sympathetic, because I know Bazel is a pain in the ass to learn, but it sounds like the author was juggling knives in a windstorm and caught a few pointy ends.

r2vcap 1y ago

Whether you use Bazel or not, this is a well-known issue with an equally well-known solution. There’s no need for such a lengthy write-up: just use a consistent sysroot across your entire build environment. If you can’t create your own sysroot image, you can simply download Chromium’s prebuilt one and configure your C++ compile rules correctly. Problem solved.

01HNNWZ0MV43FF 1y ago
Sounds like a jerb for Docker
rockwotj 1y ago
Yeah I built a custom sysroot for Redpanda (Bazel/C++/Distributed Kafka) using a really simple docker image: https://github.com/redpanda-data/redpanda/blob/dev/bazel/toolchain/Dockerfile.sysroot https://github.com/redpanda-data/redpanda/blob/dev/bazel/too... We also have an dockerfile for clang/LLVM in that repo so the whole thing is hermetic. It’s a bit of shame Bazel doesn’t come with stronger options/defaults here, because I feel like I want to reproduce this same toolchain on every C++ project with Bazel
- DriftRegion 1y ago
  Thanks for sharing. As a non-google bazel user this is quite helpful.

cryptonector 1y ago

This happens in CIs. It's happened to me on GitHub Actions. The answer is always sysroots.

Arrowmaster 1y ago
I'm in the middle of submitting PRs to multiple projects because they are compiling on ubuntu-latest and forcing a glibc 2.38 requirement. These are multiplatform projects where most or none of the devs use Linux. The first project I was able to change their workflow to build inside a 20.04 container. The other project uses tauri and it requires some very recent libraries so I don't know if an older container will work. Do you have any documentation or generic recommendations for solving these issues caused by blindly using GitHub Actions for all compilations?
- cryptonector 1y ago
  > The first project I was able to change their workflow to build inside a 20.04 container. This approach does _not_ work because you end up with the `node` that runs GitHub Actions not being able to run, certainly this will happen if you end using a sufficiently old container. > Do you have any documentation or generic recommendations for solving these issues caused by blindly using GitHub Actions for all compilations? Install these pkgs in an `ubuntu-latest` image: - debootstrap debian-archive-keyring - software-properties-common - schroot fakeroot fakechroot then - name: 'Cache sysroot' # This comes after checking out the sources because # actions/checkout@v4 cleans $PWD! id: cache-sysroot uses: actions/cache@v3 with: path: ${{ github.workspace }}/sysroot-DEBIAN_RELEASE key: sysroot-DEBIAN_RELEASE-${{ runner.os }}-${{ runner.arch }}-v1 - name: 'Setup cross-compilation sysroot' if: steps.cache-sysroot.oututs.cache-hit != 'true' run: | set -vx SYSROOT_PATH="${{ github.workspace }}/sysroot-DEBIAN_RELEASE" echo "SYSROOT_PATH=$SYSROOT_PATH" >> $GITHUB_ENV if [ ! -d sysroot-DEBIAN_RELEASE ]; then sudo debootstrap --arch=$(dpkg --print-architecture) DEBIAN_RELEASE sysroot-DEBIAN_RELEASE http://archive.ubuntu.com/ubuntu fi sudo chroot sysroot-DEBIAN_RELEASE apt-get update sudo chroot sysroot-DEBIAN_RELEASE apt-get install -y build-essential git wget curl sudo unzip zip autoconf libfreetype6-dev libcups2-dev libx11-dev libxext-dev libxrender-dev libxrandr-dev libxtst-dev libxt-dev libasound2-dev libffi-dev file binutils libfontconfig-dev sudo chroot sysroot-DEBIAN_RELEASE apt-get install -y software-properties-common sudo chroot sysroot-DEBIAN_RELEASE sudo add-apt-repository ppa:ubuntu-toolchain-r/test sudo chown -R $USER:$USER sysroot-DEBIAN_RELEASE where you replace `DEBIAN_RELEASE` with the release you want to target, and then - configure your project's build to use that sysroot. That's it. If your project does not support sysroots, make it do so. In general compilers will support sysroots, so it's just a matter of making your build configuration facility support sysroots.

setheron 1y ago

Am I the first to say "Nix solves this" Nix effectively ships a sysroot with every binary !

steeve 1y ago

This sandboxed LLVM toolchain completely solves that problem: https://github.com/cerisier/toolchains_llvm_bootstrapped https://github.com/cerisier/toolchains_llvm_bootstrapped

shaftway 1y ago

> The CI system picks it up, gets a cache hit from the developer’s build, and produces a release artifact. Why would you cache developer builds on CI?