5 ms·

Bazel is a fully reproducible and hermetic build system. A lot of painstaking work goes into it producing the exact same artifacts build after build. And that p

by jsw 4y ago

Bazel is a fully reproducible and hermetic build system. A lot of painstaking work goes into it producing the exact same artifacts build after build. And that provides some interesting properties that you can leverage for artifact caching, deployments, and CICD. We very happily runny a polyglot monorepo w/ 5+ languages, multiple architectures, with fully reproducible artifacts and deployment manifests, all deployed to almost all AWS regions on every build. We update tens of thousands of resources in every environment for every build. The fact that Bazel is creating reproducible artifacts allow us to manage this seamlessly and reliably. Clean builds take an hour+, but our GH self-hosted runners often complete commit to green build for our devs in less than a minute. The core concept of Bazel is very simple: explicitly declare the input you pass to a rule/tool and explicitly declare the output it creates. If that can click, you're half way there.

jeffbee 4y ago
Having spent a great deal of time getting bazel set up as you describe, I feel you have given readers a misleading impression. Bazel does not come out of the box that way. It uses whatever toolchain is laying around on the host, by default. It builds with system headers and links with system libraries. It looks at your environment variables. You need to do a lot of surgery on the toolchain to make it hermetic and reproducible.
- tehlike 4y ago
  Sounds like a job for docker or nix?
  - jeffbee 4y ago
    The bazel way to do it is to put the tools in an archive, refer to the archive and its checksum in your workspace, and execute the build in a completely empty sandbox.
    - hakre 4y ago
      Can you share a bit about the completely empty sandbox? Is this a build-root with it's own user and environment? Or does it build inside worktree, e.g. a subdirectory. Or can both be done?
  - hxtk 4y ago
    I can't comment on Nix, but I work on a project that uses recursive Makefiles and build containers to make the builds (almost) hermetic. I dread running builds because it takes like ten minutes to run `make all`. I spent like four hours writing Bazel build files for it and all of a sudden my clean rebuilds were taking like five minutes and my incremental builds were taking a couple of seconds. It was fantastic. Ultimately it was rejected because other people on the project hated working with bazel and weren't familiar enough when it to debug any problems. If a build didn't work, the only solution was to call me because no one else was used to interpreting the bazel errors: if a curl command failed in a script in a docker file somewhere, everyone knew what that meant. But I was the only one who would see "IOException fetching... Error 401" and immediately know that they provided the wrong password. I also consulted for another project where people were using containers as VMs, running systemd and copying in updated binaries, because they, too dreaded rebuilding the containers. Bazel, again, made it so that they could rebuild everything in a matter of seconds. That project is still happily using bazel last I checked.
  - chriswarbo 4y ago
    Docker is a red herring. The parent's statement can be modified for Docker like this: > B̶a̶z̶e̶l̶Docker uses whatever toolchain is laying around on the h̶o̶s̶t̶Internet More precisely, Docker can use exact, well-specified, cryptographically-tamper-proof environments (images); but the only way to actually specify and build such an environment is via shell scripts (in a "Dockerfile"). In practice, such scripts tend to run some other tool to do the actual specification/building, e.g. `make`, `mvn`, `cargo`, `nix`, etc. If those tools aren't reproducible, then wrapping them in Docker doesn't make them reproducible (sure we can make a snapshot, but that's basically just a dirty cache). In reality, most Dockerfiles seem to run wildly unreproducible commands, e.g. I've encountered Dockerfiles with stuff like `yum install -y pip && pip install foo`. If those tools are reproducible, then wrapping them in Docker is unnecessary. Also note that outputting a container is nothing special; most of those tools can do it (e.g. in a pinch you can have `make` run `tar` and `sha256sum` "manually")
deleted 4y ago
[deleted]
bobsomers 4y ago
> Bazel is a fully reproducible and hermetic build system. Yes, and it's very important to note that Bazel does nothing to solve the problem about having a reproducible and hermetic runtime. Even if you think you aren't linking against anything dynamically, you are probably linking against several system libraries which must be present in the same versions to get reproducible and hermetic behavior. This is solvable with Docker or exceptionally arcane Linux hackery, but it's completely missing from the Bazel messaging and it often leaves people thinking it provides more than it really does.
- hxtk 4y ago
  Others have mentioned statically linked binaries. I thought I'd mention I actually use Bazel to build most of my docker images, and unlike Docker itself, Bazel rules_docker builds reproducible (i.e., same digest) container images by default.
- ihgann 4y ago
  It’s also easily solvable with Nix (which I would hope continues to gain traction).
  - Rapzid 4y ago
    Nix, Hurd, and Haskell will walk into a bar.. Soon!
    - ihgann 4y ago
      My main OS is NixOS… and for exercise I do CrossFit. Maybe I just like cults ^_^
      - seabrookmx 4y ago
        At minimum you're self aware :)
- amscanne 4y ago
  Bazel is perfectly capable of producing static binaries, if that's what you want. It's not fair to say that does nothing to solve this problem, it simply does not mandate that the artifacts it produces are static.
matnosner 4y ago
The concept of explicitly declared inputs and outputs is awesome. Though it's closed build and the necessity to define the builds down to the compiler level makes Bazel complex and breaks some developer workflows. For this reason we are building a Bazel competitor with a less restrictive build environment which can also be applied to projects with lower than 1M line of code. https://bob.build https://bob.build

by jsw 4y ago

jeffbee 4y ago

Having spent a great deal of time getting bazel set up as you describe, I feel you have given readers a misleading impression. Bazel does not come out of the box that way. It uses whatever toolchain is laying around on the host, by default. It builds with system headers and links with system libraries. It looks at your environment variables. You need to do a lot of surgery on the toolchain to make it hermetic and reproducible.

tehlike 4y ago
Sounds like a job for docker or nix?
- jeffbee 4y ago
  The bazel way to do it is to put the tools in an archive, refer to the archive and its checksum in your workspace, and execute the build in a completely empty sandbox.
  - hakre 4y ago
    Can you share a bit about the completely empty sandbox? Is this a build-root with it's own user and environment? Or does it build inside worktree, e.g. a subdirectory. Or can both be done?
- hxtk 4y ago
  I can't comment on Nix, but I work on a project that uses recursive Makefiles and build containers to make the builds (almost) hermetic. I dread running builds because it takes like ten minutes to run `make all`. I spent like four hours writing Bazel build files for it and all of a sudden my clean rebuilds were taking like five minutes and my incremental builds were taking a couple of seconds. It was fantastic. Ultimately it was rejected because other people on the project hated working with bazel and weren't familiar enough when it to debug any problems. If a build didn't work, the only solution was to call me because no one else was used to interpreting the bazel errors: if a curl command failed in a script in a docker file somewhere, everyone knew what that meant. But I was the only one who would see "IOException fetching... Error 401" and immediately know that they provided the wrong password. I also consulted for another project where people were using containers as VMs, running systemd and copying in updated binaries, because they, too dreaded rebuilding the containers. Bazel, again, made it so that they could rebuild everything in a matter of seconds. That project is still happily using bazel last I checked.
- chriswarbo 4y ago
  Docker is a red herring. The parent's statement can be modified for Docker like this: > B̶a̶z̶e̶l̶Docker uses whatever toolchain is laying around on the h̶o̶s̶t̶Internet More precisely, Docker can use exact, well-specified, cryptographically-tamper-proof environments (images); but the only way to actually specify and build such an environment is via shell scripts (in a "Dockerfile"). In practice, such scripts tend to run some other tool to do the actual specification/building, e.g. `make`, `mvn`, `cargo`, `nix`, etc. If those tools aren't reproducible, then wrapping them in Docker doesn't make them reproducible (sure we can make a snapshot, but that's basically just a dirty cache). In reality, most Dockerfiles seem to run wildly unreproducible commands, e.g. I've encountered Dockerfiles with stuff like `yum install -y pip && pip install foo`. If those tools are reproducible, then wrapping them in Docker is unnecessary. Also note that outputting a container is nothing special; most of those tools can do it (e.g. in a pinch you can have `make` run `tar` and `sha256sum` "manually")

deleted 4y ago

[deleted]

bobsomers 4y ago

> Bazel is a fully reproducible and hermetic build system. Yes, and it's very important to note that Bazel does nothing to solve the problem about having a reproducible and hermetic runtime. Even if you think you aren't linking against anything dynamically, you are probably linking against several system libraries which must be present in the same versions to get reproducible and hermetic behavior. This is solvable with Docker or exceptionally arcane Linux hackery, but it's completely missing from the Bazel messaging and it often leaves people thinking it provides more than it really does.

hxtk 4y ago
Others have mentioned statically linked binaries. I thought I'd mention I actually use Bazel to build most of my docker images, and unlike Docker itself, Bazel rules_docker builds reproducible (i.e., same digest) container images by default.
ihgann 4y ago
It’s also easily solvable with Nix (which I would hope continues to gain traction).
- Rapzid 4y ago
  Nix, Hurd, and Haskell will walk into a bar.. Soon!
  - ihgann 4y ago
    My main OS is NixOS… and for exercise I do CrossFit. Maybe I just like cults ^_^
    - seabrookmx 4y ago
      At minimum you're self aware :)
amscanne 4y ago
Bazel is perfectly capable of producing static binaries, if that's what you want. It's not fair to say that does nothing to solve this problem, it simply does not mandate that the artifacts it produces are static.

matnosner 4y ago

The concept of explicitly declared inputs and outputs is awesome. Though it's closed build and the necessity to define the builds down to the compiler level makes Bazel complex and breaks some developer workflows. For this reason we are building a Bazel competitor with a less restrictive build environment which can also be applied to projects with lower than 1M line of code. https://bob.build https://bob.build