4 ms·

One other hidden Bazel trap that I've seen is for companies to migrate a large codebase to Bazel, but then to rely on OS-provided tools and libraries. Commonly,

One other hidden Bazel trap that I've seen is for companies to migrate a large codebase to Bazel, but then to rely on OS-provided tools and libraries. Commonly, this gets paired with a glib answer like "it's fine, we build from inside of a Docker container". But I've never seen that Docker image linked into Bazel's dependency resolver, or the compose scripts used to launch the container. This has the following effects: 1. There are unexpressed package/tool dependencies. 2. Across a large organization, Bazel's reproducibility guarantees go out the window. 3. Developers can't just clone the repo and start using Bazel. Instead, they have to pull down some pinned Docker image, or build it themselves and lose reproducibility. 4. This effectively poisons the cache whenever the Docker image is updated or rebuilt. If using a shared remote cache, it can be a major issue. If an organization isn't big enough to vendor every single tool dependency, shared library, etc (which basically requires building out an OS distribution in Bazel), what's the right way to approach this problem?

setheron 2y ago
Docker did us wonders but also set us back in many ways (same view point about the proliferation of development on MacOS). I've lately been thinking why isn't there a public vendored third_party set to use. If we can agree living at HEAD is preferred having a communal vendored third_party seems like a clear win.
- klodolph 2y ago
  > If we can agree living at HEAD is preferred having a communal vendored third_party seems like a clear win. “It has been 0 days since a minor version update in a third-party library has broken my code.” This is just a fiendishly difficult problem to solve.
klodolph 2y ago
> If an organization isn't big enough to vendor every single tool dependency, shared library, etc (which basically requires building out an OS distribution in Bazel), what's the right way to approach this problem? Here are some ways you can approach this problem: If you have a bunch of shared tools & libraries and you don’t want to vendor the whole packages, you can vendor the artifacts. You can do this piecemeal or as a big chunk. Basically, instead of a Docker image containing your compiler or whatever, you have a tarball. You make Bazel responsible for downloading the tarball. Or maybe it’s several tarballs, it doesn’t matter. Bazel is actually pretty good at this. Another approach is to combine Bazel with something else reproducible, like Nix. For now, I’d suggest taking a very basic approach, the most simple approach, which is to create an environment using Nix (containing your toolchain, third-party libraries, and Bazel) and then building your code inside that environment using Bazel. You can use the local_repository() rule to access the Nix environment. Bazel + Nix is a really good idea, but maybe now is not the time to dive into that. When I’ve investigated the Bazel + Nix combination, it looks like neither system is exactly stable / mature enough yet. Bazel just recently launched bzlmod and the Bazel ecosystem is shifting to use bzlmod everywhere. Nix is shifting to flakes but there are some pains there too (especially around documentation, but there are also some things that don’t quite work with flakes yet). So in the future, you could do something kind of, well, cursed, where you have a Bazel and Nix lasagna. You use Nix to run Bazel, and then you use Nix packages as dependencies of your Bazel build. In theory, this could give you the best of both worlds—you get the wonderful fine-grained dependencies of Bazel, the rich build system, the super fast performance. And then you get access to reproducible builds of all sorts of third-party dependencies through Nix. In practice, you need to become something of an expert in both Bazel and Nix in order to do this. As a final option, depending on the languages you are using, you may be happy with just plain bzlmod. Like, Go integration with Bazel is damn solid. You don’t need to vendor anything, you can just keep using go.mod, and Bazel will deal with it (with some help from Gazelle). Bazel will even download the Go compiler for you, without any additional setup. A lot of other languages work the same way—it’s just that C and C++ are notable exceptions, where Bazel just grabs the system compiler by default, and package management for C and C++ is complete chaos.
- nrclark 2y ago
  Is there any good way to make some Bazel targets hermetic, and some non-hermetic? I'd love to be able to set my build up with something kind of like: 1. Non-isolated Bazel target that produces a container image. 2. All other targets are isolated, and build inside of a container created from the new image.
  - klodolph 2y ago
    This is a good question, but it took me a few tries to write a satisfactory answer. A hermetic build is one where the environment can change but the build outputs stay the same. This lets you aggressively and pervasively use remote caches, which can give massive performance benefits to large and complicated builds. I’m sure you know this, I’m just going through so I can clarify it in my mind. If your Docker image depends on the environment but your final build does not, then congratulations, you’ve made a hermetic build. But I don’t see how building the Docker image in Bazel is helpful here. The more likely result here, I think, is that you have a build which is still non-hermetic. The inner build steps are isolated from your current environment, but they are not isolated from the environment used to create the Docker image. As a matter of practical advice, try creating your Docker image in Nix. Nix is very good at creating Docker images hermetically and has a large number of third-party packages and toolchains available. Bazel is very good at fast builds and fine-grained dependencies. The easiest, simplest thing you can do with this combination of tools is to have Nix create a Docker image that contains Bazel and all of your build dependencies that aren’t vendored into the Bazel tree. You can create hermetic builds by carefully designing your build rules so they aren’t affected by the environment. But it is a lot easier and more reliable to just stamp SHA-256 checksums on chunks of your environment, when that option is available.
rtpg 2y ago
I will say that it's _extremely easy_ to just list `Dockerfile` in your output dependencies if it matters enough for you to care. Stuff from apt/etc as well? put the install script in the dependencies. Really care about the version numbers? Add a test that just calls `tool -v` and compares the output to some fixed number. Add a big comment. Is it perfect? No! Is it hermetic! Hell no! But is it going to catch a hell of a lot of stuff and get you fast tests along the way? Yes! The biggest point with all CI/CD is to get as much value as you can from these systems, while introducing the least amount of friction. And velocity gained from these systems mean that if you find an issue, you can fix it quickly. It's an amazing feedback loop, and leaning into that feedback loop is way more valuable than writing `gettext` compilation scripts.
- zelphirkalt 2y ago
  But what does installing things via apt have to do with reproducibility? Does apt have some way to specify lock files or hashsums of the packages it is supposed to install? Or do you mean to pin version numbers of system packages and rely on that? Otherwise apt would be the point where the guarantees go out of the window already.
  - seabass-labrax 2y ago
    Debian can pin packages to certain versions by their numbers (see dpkg(1), '--set-selections') and it does verify package integrity. I can't think of any way to pin a package to a hash like with Bazel or Nix, but the expectation is that packages are not changed after publication in dpkg repositories - and for Debian itself, that expectation is a strictly-followed rule. Therefore I would trust package pinning to work, but it's not quite as straightforward for the end-user as unique package hashes as identifiers.

setheron 2y ago

Docker did us wonders but also set us back in many ways (same view point about the proliferation of development on MacOS). I've lately been thinking why isn't there a public vendored third_party set to use. If we can agree living at HEAD is preferred having a communal vendored third_party seems like a clear win.

klodolph 2y ago
> If we can agree living at HEAD is preferred having a communal vendored third_party seems like a clear win. “It has been 0 days since a minor version update in a third-party library has broken my code.” This is just a fiendishly difficult problem to solve.

klodolph 2y ago

> If an organization isn't big enough to vendor every single tool dependency, shared library, etc (which basically requires building out an OS distribution in Bazel), what's the right way to approach this problem? Here are some ways you can approach this problem: If you have a bunch of shared tools & libraries and you don’t want to vendor the whole packages, you can vendor the artifacts. You can do this piecemeal or as a big chunk. Basically, instead of a Docker image containing your compiler or whatever, you have a tarball. You make Bazel responsible for downloading the tarball. Or maybe it’s several tarballs, it doesn’t matter. Bazel is actually pretty good at this. Another approach is to combine Bazel with something else reproducible, like Nix. For now, I’d suggest taking a very basic approach, the most simple approach, which is to create an environment using Nix (containing your toolchain, third-party libraries, and Bazel) and then building your code inside that environment using Bazel. You can use the local_repository() rule to access the Nix environment. Bazel + Nix is a really good idea, but maybe now is not the time to dive into that. When I’ve investigated the Bazel + Nix combination, it looks like neither system is exactly stable / mature enough yet. Bazel just recently launched bzlmod and the Bazel ecosystem is shifting to use bzlmod everywhere. Nix is shifting to flakes but there are some pains there too (especially around documentation, but there are also some things that don’t quite work with flakes yet). So in the future, you could do something kind of, well, cursed, where you have a Bazel and Nix lasagna. You use Nix to run Bazel, and then you use Nix packages as dependencies of your Bazel build. In theory, this could give you the best of both worlds—you get the wonderful fine-grained dependencies of Bazel, the rich build system, the super fast performance. And then you get access to reproducible builds of all sorts of third-party dependencies through Nix. In practice, you need to become something of an expert in both Bazel and Nix in order to do this. As a final option, depending on the languages you are using, you may be happy with just plain bzlmod. Like, Go integration with Bazel is damn solid. You don’t need to vendor anything, you can just keep using go.mod, and Bazel will deal with it (with some help from Gazelle). Bazel will even download the Go compiler for you, without any additional setup. A lot of other languages work the same way—it’s just that C and C++ are notable exceptions, where Bazel just grabs the system compiler by default, and package management for C and C++ is complete chaos.

nrclark 2y ago
Is there any good way to make some Bazel targets hermetic, and some non-hermetic? I'd love to be able to set my build up with something kind of like: 1. Non-isolated Bazel target that produces a container image. 2. All other targets are isolated, and build inside of a container created from the new image.
- klodolph 2y ago
  This is a good question, but it took me a few tries to write a satisfactory answer. A hermetic build is one where the environment can change but the build outputs stay the same. This lets you aggressively and pervasively use remote caches, which can give massive performance benefits to large and complicated builds. I’m sure you know this, I’m just going through so I can clarify it in my mind. If your Docker image depends on the environment but your final build does not, then congratulations, you’ve made a hermetic build. But I don’t see how building the Docker image in Bazel is helpful here. The more likely result here, I think, is that you have a build which is still non-hermetic. The inner build steps are isolated from your current environment, but they are not isolated from the environment used to create the Docker image. As a matter of practical advice, try creating your Docker image in Nix. Nix is very good at creating Docker images hermetically and has a large number of third-party packages and toolchains available. Bazel is very good at fast builds and fine-grained dependencies. The easiest, simplest thing you can do with this combination of tools is to have Nix create a Docker image that contains Bazel and all of your build dependencies that aren’t vendored into the Bazel tree. You can create hermetic builds by carefully designing your build rules so they aren’t affected by the environment. But it is a lot easier and more reliable to just stamp SHA-256 checksums on chunks of your environment, when that option is available.

rtpg 2y ago

I will say that it's _extremely easy_ to just list `Dockerfile` in your output dependencies if it matters enough for you to care. Stuff from apt/etc as well? put the install script in the dependencies. Really care about the version numbers? Add a test that just calls `tool -v` and compares the output to some fixed number. Add a big comment. Is it perfect? No! Is it hermetic! Hell no! But is it going to catch a hell of a lot of stuff and get you fast tests along the way? Yes! The biggest point with all CI/CD is to get as much value as you can from these systems, while introducing the least amount of friction. And velocity gained from these systems mean that if you find an issue, you can fix it quickly. It's an amazing feedback loop, and leaning into that feedback loop is way more valuable than writing `gettext` compilation scripts.

zelphirkalt 2y ago
But what does installing things via apt have to do with reproducibility? Does apt have some way to specify lock files or hashsums of the packages it is supposed to install? Or do you mean to pin version numbers of system packages and rely on that? Otherwise apt would be the point where the guarantees go out of the window already.
- seabass-labrax 2y ago
  Debian can pin packages to certain versions by their numbers (see dpkg(1), '--set-selections') and it does verify package integrity. I can't think of any way to pin a package to a hash like with Bazel or Nix, but the expectation is that packages are not changed after publication in dpkg repositories - and for Debian itself, that expectation is a strictly-followed rule. Therefore I would trust package pinning to work, but it's not quite as straightforward for the end-user as unique package hashes as identifiers.