3 ms·

> If an organization isn't big enough to vendor every single tool dependency, shared library, etc (which basically requires building out an OS distribution in B

> If an organization isn't big enough to vendor every single tool dependency, shared library, etc (which basically requires building out an OS distribution in Bazel), what's the right way to approach this problem? Here are some ways you can approach this problem: If you have a bunch of shared tools & libraries and you don’t want to vendor the whole packages, you can vendor the artifacts. You can do this piecemeal or as a big chunk. Basically, instead of a Docker image containing your compiler or whatever, you have a tarball. You make Bazel responsible for downloading the tarball. Or maybe it’s several tarballs, it doesn’t matter. Bazel is actually pretty good at this. Another approach is to combine Bazel with something else reproducible, like Nix. For now, I’d suggest taking a very basic approach, the most simple approach, which is to create an environment using Nix (containing your toolchain, third-party libraries, and Bazel) and then building your code inside that environment using Bazel. You can use the local_repository() rule to access the Nix environment. Bazel + Nix is a really good idea, but maybe now is not the time to dive into that. When I’ve investigated the Bazel + Nix combination, it looks like neither system is exactly stable / mature enough yet. Bazel just recently launched bzlmod and the Bazel ecosystem is shifting to use bzlmod everywhere. Nix is shifting to flakes but there are some pains there too (especially around documentation, but there are also some things that don’t quite work with flakes yet). So in the future, you could do something kind of, well, cursed, where you have a Bazel and Nix lasagna. You use Nix to run Bazel, and then you use Nix packages as dependencies of your Bazel build. In theory, this could give you the best of both worlds—you get the wonderful fine-grained dependencies of Bazel, the rich build system, the super fast performance. And then you get access to reproducible builds of all sorts of third-party dependencies through Nix. In practice, you need to become something of an expert in both Bazel and Nix in order to do this. As a final option, depending on the languages you are using, you may be happy with just plain bzlmod. Like, Go integration with Bazel is damn solid. You don’t need to vendor anything, you can just keep using go.mod, and Bazel will deal with it (with some help from Gazelle). Bazel will even download the Go compiler for you, without any additional setup. A lot of other languages work the same way—it’s just that C and C++ are notable exceptions, where Bazel just grabs the system compiler by default, and package management for C and C++ is complete chaos.

nrclark 2y ago
Is there any good way to make some Bazel targets hermetic, and some non-hermetic? I'd love to be able to set my build up with something kind of like: 1. Non-isolated Bazel target that produces a container image. 2. All other targets are isolated, and build inside of a container created from the new image.
- klodolph 2y ago
  This is a good question, but it took me a few tries to write a satisfactory answer. A hermetic build is one where the environment can change but the build outputs stay the same. This lets you aggressively and pervasively use remote caches, which can give massive performance benefits to large and complicated builds. I’m sure you know this, I’m just going through so I can clarify it in my mind. If your Docker image depends on the environment but your final build does not, then congratulations, you’ve made a hermetic build. But I don’t see how building the Docker image in Bazel is helpful here. The more likely result here, I think, is that you have a build which is still non-hermetic. The inner build steps are isolated from your current environment, but they are not isolated from the environment used to create the Docker image. As a matter of practical advice, try creating your Docker image in Nix. Nix is very good at creating Docker images hermetically and has a large number of third-party packages and toolchains available. Bazel is very good at fast builds and fine-grained dependencies. The easiest, simplest thing you can do with this combination of tools is to have Nix create a Docker image that contains Bazel and all of your build dependencies that aren’t vendored into the Bazel tree. You can create hermetic builds by carefully designing your build rules so they aren’t affected by the environment. But it is a lot easier and more reliable to just stamp SHA-256 checksums on chunks of your environment, when that option is available.

nrclark 2y ago

Is there any good way to make some Bazel targets hermetic, and some non-hermetic? I'd love to be able to set my build up with something kind of like: 1. Non-isolated Bazel target that produces a container image. 2. All other targets are isolated, and build inside of a container created from the new image.

klodolph 2y ago
This is a good question, but it took me a few tries to write a satisfactory answer. A hermetic build is one where the environment can change but the build outputs stay the same. This lets you aggressively and pervasively use remote caches, which can give massive performance benefits to large and complicated builds. I’m sure you know this, I’m just going through so I can clarify it in my mind. If your Docker image depends on the environment but your final build does not, then congratulations, you’ve made a hermetic build. But I don’t see how building the Docker image in Bazel is helpful here. The more likely result here, I think, is that you have a build which is still non-hermetic. The inner build steps are isolated from your current environment, but they are not isolated from the environment used to create the Docker image. As a matter of practical advice, try creating your Docker image in Nix. Nix is very good at creating Docker images hermetically and has a large number of third-party packages and toolchains available. Bazel is very good at fast builds and fine-grained dependencies. The easiest, simplest thing you can do with this combination of tools is to have Nix create a Docker image that contains Bazel and all of your build dependencies that aren’t vendored into the Bazel tree. You can create hermetic builds by carefully designing your build rules so they aren’t affected by the environment. But it is a lot easier and more reliable to just stamp SHA-256 checksums on chunks of your environment, when that option is available.