4 ms·

> It is closer to being fully reproducible than most other build systems (including Bazel). How so? Bazel produces the same results for the same inputs.

by sa46 2y ago

jchw 2y ago
Bazel doesn't guarantee bit-exact outputs, but also Bazel doesn't guarantee pure builds. It does have a sandbox that prevents some impurities, but for example it doesn't prevent things from going out to the network, or even accessing files from anywhere in the filesystem, if you use absolute paths. (Although, on Linux at least, Bazel does prevent you from modifying files outside of the sandbox directory.) The Nix sandbox does completely obscure the host filesystem and limit network access to processes that can produce a bit-exact output only. (Bazel also obviously uses the system compilers and headers. Nix does not.)
- dijit 2y ago
  Uh, Either my understanding of Bazel is wrong, or everything you wrote is wrong. Bazel absolutely prevents network access and filesystem access (reads) from builds. (only permitting explicit network includes from the WORKSPACE file, and access to files explicitly depended on in the BUILD files). Maybe you can write some “rules_” for languages that violate this, but it is designed purposely to be hermetic and bit-perfect reproducible. EDIT: From the FAQ[0]: > Will Bazel make my builds reproducible automatically? > For Java and C++ binaries, yes, assuming you do not change the toolchain. The issues with Docker's style of "reproducible" (meaning.. consistent environment; are also outlined in the same FAQ[1] > Doesn’t Docker solve the reproducibility problems? > Docker does not address reproducibility with regard to changes in the source code. Running Make with an imperfectly written Makefile inside a Docker container can still yield unpredictable results. [0]: https://bazel.build/about/faq#will_bazel_make_my_builds_reproducible_automatically https://bazel.build/about/faq#will_bazel_make_my_builds_repr... [1]: https://bazel.build/about/faq#doesn’t_docker_solve_the_reproducibility_problems https://bazel.build/about/faq#doesn’t_docker_solve_the_repro...
  - valcron1000 2y ago
    I'm not familiar with Bazel at all so this might be obvious, but does Bazel check that the files listed in the BUILD file are the "right ones" (ex. through a checksum), and if so, is this always enforced (that is, this behavior cannot be disabled)?
    - dijit 2y ago
      The contents of files are basically hashed, if the contents don't change of the file listed for a target then no change will happen, even if you modify metadata of the file (like last modified time by `touch` or so on.) Bazel is really sophisticated and I'd be lying if I said I understood it well, but I have spent time looking at it.
  - keithwinstein 2y ago
    I think you're both right in a sense. Bazel doesn't (in general) prevent filesystem access, e.g. to library headers in /usr/include. If those headers change (maybe because a Debian package got upgraded or whatever), Bazel won't know it has to invalidate the build cache. I think the FAQ is still technically correct because upgrading the Debian package for a random library dependency counts as "chang[ing] the toolchain" in this context. But I don't think you'd call it hermetic by default. Check out the previous discussion at https://news.ycombinator.com/item?id=23184843 https://news.ycombinator.com/item?id=23184843 and below: > Under the hood there's a default auto-configured toolchain that finds whatever is installed locally in the system. Since it has no way of knowing what files an arbitrary "cc" might depend on, you lose hermeticity by using it.
  - amarshall 2y ago
    AFAIK Bazel does not use the sandbox by default. Last time I experimented with it, the sandbox had some problematic holes, but I don’t remember exactly what, and it’s been a few years. The very doc you link hints at that, while also giving many caveats where the build will become non-reproducible. So it boils down to “yes, but only if you configure it correctly and do things right”.
    - 2y ago
- gf000 2y ago
  I think talking about sandboxes is missing a point a bit. It's an important constituent, but only complete OS-emulation with deterministic scheduling could (at a huge overhead) actually result in bit-by-bit reproducible artifacts with arbitrary build steps. There are an endless source of impurities/randomness and most compilers haven't historically cared much about this.
  - jchw 2y ago
    The point I'm making is that neither Bazel nor Nix do that. However, sandboxing is still relevant, because if you still have impurities leaking from outside the closure of the build, you have bigger fish to fry than non-deterministic builds. That all said, in practice, many of the cases where Nixpkgs builds are not deterministic are actually fairly trivial. Despite not being a specific goal necessarily, compilers are more deterministic than not, and in practice the sources of non-determinism are fewer than you'd think. Case in point, I'm pretty sure the vast majority of Nixpkgs packages that are bit-for-bit reproducible just kind of are by accident, because nothing in the build is actually non-deterministic. Many of the cases of non-deterministic builds are fairly trivial, such as things just linking in different orders depending on scheduling. Running everything under a deterministic VM would probably be too slow and/or cumbersome, so I think Nix is the best it's going to get.
    - gf000 2y ago
      Sandboxing is relevant, but nix does that by default, so no difference here. Nonetheless, I agree that Nix does the optimum here, full-on emulation would be prohibitively expensive.
      - jchw 2y ago
        You know, though, it would probably be possible to develop a pretty fast "deterministic" execution environment if you just limit execution to a single thread, still not resorting to full emulation. You'd still have to deal with differences between CPUs, but it would probably not be nearly as big of an issue. And it would slow down builds, but on the other hand, you can do a lot of them in parallel. This could be pretty interesting if you combined it with trying to integrate build system output directly into the Nix DAG, because then you could get back some of the intra-build parallelism, too. Wouldn't be applicable for Nixpkgs since it would require ugly IFD hacks, but might be interesting for a development setup. Perhaps it's an area worth researching.
- paulddraper 2y ago
  > Bazel also obviously uses the system compilers and headers. Nix does not. Bazel allows hermetic toolchains, and uses it for most languages: Java, Python, Go, Rust, Node.js, etc. You can do the same for C++, but Bazel doesn't provide that out-of-the-box. [1] Bazel sandboxing can restrict system access on Linux with --experimental_use_hermetic_linux_sandbox and --sandbox_add_mount_pair. [2] Every "reproducible builds" discussion requires an understand of what is permitted to vary. E.g. Neither Nix nor Bazel attempts to make build products the same for x86 host environments vs ARM host environments. Bazel is less aggressive than Nix in that it does not (by default) attempt to make build products the same for different host C++ compilers. [1] https://github.com/bazelbuild/bazel/discussions/18332 https://github.com/bazelbuild/bazel/discussions/18332 [2] https://bazel.build/reference/command-line-reference#flag--experimental_use_hermetic_linux_sandbox https://bazel.build/reference/command-line-reference#flag--e...
gf000 2y ago
No, most compilers are not themselves reproducible, even within very restrictive sandboxes (e.g. they may do some work concurrently and collect the results based on when it completes, then build on top of that. If they don't add a timing-insensitive sorting step, the resulting binary will (assuming no bugs) be functionally equivalent, but may not be bit-by-by equal), and a build tool can only do so much.

> It is closer to being fully reproducible than most other build systems (including Bazel). How so? Bazel produces the same results for the same inputs.

by sa46 2y ago

> It is closer to being fully reproducible than most other build systems (including Bazel). How so? Bazel produces the same results for the same inputs.

jchw 2y ago

Bazel doesn't guarantee bit-exact outputs, but also Bazel doesn't guarantee pure builds. It does have a sandbox that prevents some impurities, but for example it doesn't prevent things from going out to the network, or even accessing files from anywhere in the filesystem, if you use absolute paths. (Although, on Linux at least, Bazel does prevent you from modifying files outside of the sandbox directory.) The Nix sandbox does completely obscure the host filesystem and limit network access to processes that can produce a bit-exact output only. (Bazel also obviously uses the system compilers and headers. Nix does not.)

dijit 2y ago
Uh, Either my understanding of Bazel is wrong, or everything you wrote is wrong. Bazel absolutely prevents network access and filesystem access (reads) from builds. (only permitting explicit network includes from the WORKSPACE file, and access to files explicitly depended on in the BUILD files). Maybe you can write some “rules_” for languages that violate this, but it is designed purposely to be hermetic and bit-perfect reproducible. EDIT: From the FAQ[0]: > Will Bazel make my builds reproducible automatically? > For Java and C++ binaries, yes, assuming you do not change the toolchain. The issues with Docker's style of "reproducible" (meaning.. consistent environment; are also outlined in the same FAQ[1] > Doesn’t Docker solve the reproducibility problems? > Docker does not address reproducibility with regard to changes in the source code. Running Make with an imperfectly written Makefile inside a Docker container can still yield unpredictable results. [0]: https://bazel.build/about/faq#will_bazel_make_my_builds_reproducible_automatically https://bazel.build/about/faq#will_bazel_make_my_builds_repr... [1]: https://bazel.build/about/faq#doesn’t_docker_solve_the_reproducibility_problems https://bazel.build/about/faq#doesn’t_docker_solve_the_repro...
- valcron1000 2y ago
  I'm not familiar with Bazel at all so this might be obvious, but does Bazel check that the files listed in the BUILD file are the "right ones" (ex. through a checksum), and if so, is this always enforced (that is, this behavior cannot be disabled)?
  - dijit 2y ago
    The contents of files are basically hashed, if the contents don't change of the file listed for a target then no change will happen, even if you modify metadata of the file (like last modified time by `touch` or so on.) Bazel is really sophisticated and I'd be lying if I said I understood it well, but I have spent time looking at it.
- keithwinstein 2y ago
  I think you're both right in a sense. Bazel doesn't (in general) prevent filesystem access, e.g. to library headers in /usr/include. If those headers change (maybe because a Debian package got upgraded or whatever), Bazel won't know it has to invalidate the build cache. I think the FAQ is still technically correct because upgrading the Debian package for a random library dependency counts as "chang[ing] the toolchain" in this context. But I don't think you'd call it hermetic by default. Check out the previous discussion at https://news.ycombinator.com/item?id=23184843 https://news.ycombinator.com/item?id=23184843 and below: > Under the hood there's a default auto-configured toolchain that finds whatever is installed locally in the system. Since it has no way of knowing what files an arbitrary "cc" might depend on, you lose hermeticity by using it.
- amarshall 2y ago
  AFAIK Bazel does not use the sandbox by default. Last time I experimented with it, the sandbox had some problematic holes, but I don’t remember exactly what, and it’s been a few years. The very doc you link hints at that, while also giving many caveats where the build will become non-reproducible. So it boils down to “yes, but only if you configure it correctly and do things right”.
  - 2y ago
gf000 2y ago
I think talking about sandboxes is missing a point a bit. It's an important constituent, but only complete OS-emulation with deterministic scheduling could (at a huge overhead) actually result in bit-by-bit reproducible artifacts with arbitrary build steps. There are an endless source of impurities/randomness and most compilers haven't historically cared much about this.
- jchw 2y ago
  The point I'm making is that neither Bazel nor Nix do that. However, sandboxing is still relevant, because if you still have impurities leaking from outside the closure of the build, you have bigger fish to fry than non-deterministic builds. That all said, in practice, many of the cases where Nixpkgs builds are not deterministic are actually fairly trivial. Despite not being a specific goal necessarily, compilers are more deterministic than not, and in practice the sources of non-determinism are fewer than you'd think. Case in point, I'm pretty sure the vast majority of Nixpkgs packages that are bit-for-bit reproducible just kind of are by accident, because nothing in the build is actually non-deterministic. Many of the cases of non-deterministic builds are fairly trivial, such as things just linking in different orders depending on scheduling. Running everything under a deterministic VM would probably be too slow and/or cumbersome, so I think Nix is the best it's going to get.
  - gf000 2y ago
    Sandboxing is relevant, but nix does that by default, so no difference here. Nonetheless, I agree that Nix does the optimum here, full-on emulation would be prohibitively expensive.
    - jchw 2y ago
      You know, though, it would probably be possible to develop a pretty fast "deterministic" execution environment if you just limit execution to a single thread, still not resorting to full emulation. You'd still have to deal with differences between CPUs, but it would probably not be nearly as big of an issue. And it would slow down builds, but on the other hand, you can do a lot of them in parallel. This could be pretty interesting if you combined it with trying to integrate build system output directly into the Nix DAG, because then you could get back some of the intra-build parallelism, too. Wouldn't be applicable for Nixpkgs since it would require ugly IFD hacks, but might be interesting for a development setup. Perhaps it's an area worth researching.
paulddraper 2y ago
> Bazel also obviously uses the system compilers and headers. Nix does not. Bazel allows hermetic toolchains, and uses it for most languages: Java, Python, Go, Rust, Node.js, etc. You can do the same for C++, but Bazel doesn't provide that out-of-the-box. [1] Bazel sandboxing can restrict system access on Linux with --experimental_use_hermetic_linux_sandbox and --sandbox_add_mount_pair. [2] Every "reproducible builds" discussion requires an understand of what is permitted to vary. E.g. Neither Nix nor Bazel attempts to make build products the same for x86 host environments vs ARM host environments. Bazel is less aggressive than Nix in that it does not (by default) attempt to make build products the same for different host C++ compilers. [1] https://github.com/bazelbuild/bazel/discussions/18332 https://github.com/bazelbuild/bazel/discussions/18332 [2] https://bazel.build/reference/command-line-reference#flag--experimental_use_hermetic_linux_sandbox https://bazel.build/reference/command-line-reference#flag--e...

gf000 2y ago

No, most compilers are not themselves reproducible, even within very restrictive sandboxes (e.g. they may do some work concurrently and collect the results based on when it completes, then build on top of that. If they don't add a timing-insensitive sorting step, the resulting binary will (assuming no bugs) be functionally equivalent, but may not be bit-by-by equal), and a build tool can only do so much.