7 ms·

I think this debate comes down to exactly what "reproducible" means. Nix doesn't give bit-exact reproducibility, but it does give reproducible environments, by

by jchw 2y ago

I think this debate comes down to exactly what "reproducible" means. Nix doesn't give bit-exact reproducibility, but it does give reproducible environments, by ensuring that the inputs are always bit-exact. It is closer to being fully reproducible than most other build systems (including Bazel) -- but because it can only reasonably ensure that the inputs are exact, it's still necessary for the build processes themselves to be fully deterministic to get end-to-end bit-exactness. Nix on its own doesn't fully resolve supply chain concerns about binaries, but it can provide answers to a myriad of other problems. I think most people like Nix reproducibility, and it is marketed as such, for the sake of development: life is much easier when you know for sure you have the exact same version of each dependency, in the exact same configuration. A build on one machine may not be bit-exact to a build on another machine, but it will be exactly the same source code all the way down. The quest to get every build process to be deterministic is definitely a bigger problem and it will never be solved for all of Nixpkgs. NixOS does have a reproducibility project[1], and some non-trivial amount of NixOS actually is properly reproducible, but the observation that Nixpkgs is too vast is definitely spot-on, especially because in most cases the real issues lie upstream. (and carrying patches for reproducibility is possible, but it adds even more maintainer burden.) [1]: https://reproducible.nixos.org/ https://reproducible.nixos.org/

sa46 2y ago
> It is closer to being fully reproducible than most other build systems (including Bazel). How so? Bazel produces the same results for the same inputs.
- jchw 2y ago
  Bazel doesn't guarantee bit-exact outputs, but also Bazel doesn't guarantee pure builds. It does have a sandbox that prevents some impurities, but for example it doesn't prevent things from going out to the network, or even accessing files from anywhere in the filesystem, if you use absolute paths. (Although, on Linux at least, Bazel does prevent you from modifying files outside of the sandbox directory.) The Nix sandbox does completely obscure the host filesystem and limit network access to processes that can produce a bit-exact output only. (Bazel also obviously uses the system compilers and headers. Nix does not.)
  - dijit 2y ago
    Uh, Either my understanding of Bazel is wrong, or everything you wrote is wrong. Bazel absolutely prevents network access and filesystem access (reads) from builds. (only permitting explicit network includes from the WORKSPACE file, and access to files explicitly depended on in the BUILD files). Maybe you can write some “rules_” for languages that violate this, but it is designed purposely to be hermetic and bit-perfect reproducible. EDIT: From the FAQ[0]: > Will Bazel make my builds reproducible automatically? > For Java and C++ binaries, yes, assuming you do not change the toolchain. The issues with Docker's style of "reproducible" (meaning.. consistent environment; are also outlined in the same FAQ[1] > Doesn’t Docker solve the reproducibility problems? > Docker does not address reproducibility with regard to changes in the source code. Running Make with an imperfectly written Makefile inside a Docker container can still yield unpredictable results. [0]: https://bazel.build/about/faq#will_bazel_make_my_builds_reproducible_automatically https://bazel.build/about/faq#will_bazel_make_my_builds_repr... [1]: https://bazel.build/about/faq#doesn’t_docker_solve_the_reproducibility_problems https://bazel.build/about/faq#doesn’t_docker_solve_the_repro...
    - valcron1000 2y ago
      I'm not familiar with Bazel at all so this might be obvious, but does Bazel check that the files listed in the BUILD file are the "right ones" (ex. through a checksum), and if so, is this always enforced (that is, this behavior cannot be disabled)?
      - dijit 2y ago
        The contents of files are basically hashed, if the contents don't change of the file listed for a target then no change will happen, even if you modify metadata of the file (like last modified time by `touch` or so on.) Bazel is really sophisticated and I'd be lying if I said I understood it well, but I have spent time looking at it.
    - keithwinstein 2y ago
      I think you're both right in a sense. Bazel doesn't (in general) prevent filesystem access, e.g. to library headers in /usr/include. If those headers change (maybe because a Debian package got upgraded or whatever), Bazel won't know it has to invalidate the build cache. I think the FAQ is still technically correct because upgrading the Debian package for a random library dependency counts as "chang[ing] the toolchain" in this context. But I don't think you'd call it hermetic by default. Check out the previous discussion at https://news.ycombinator.com/item?id=23184843 https://news.ycombinator.com/item?id=23184843 and below: > Under the hood there's a default auto-configured toolchain that finds whatever is installed locally in the system. Since it has no way of knowing what files an arbitrary "cc" might depend on, you lose hermeticity by using it.
- gf000 2y ago
  No, most compilers are not themselves reproducible, even within very restrictive sandboxes (e.g. they may do some work concurrently and collect the results based on when it completes, then build on top of that. If they don't add a timing-insensitive sorting step, the resulting binary will (assuming no bugs) be functionally equivalent, but may not be bit-by-by equal), and a build tool can only do so much.
matrss 2y ago
> The quest to get every build process to be deterministic [...] will never be solved for all of Nixpkgs. Not least because of unfree and/or binary-blob packages that can't be reproducible because they don't even build anything. As much as Guix' strict FOSS and build-from-source policy can be an annoyance, it is a necessary precondition to achieve full reproducibility from source, i.e. the full-source bootstrap.
- jchw 2y ago
  Nixpkgs provides license[1] and source provenance[2] information. For legal reasons, Nix also defaults to not evaluating unfree packages. Not packaging them at all, though, doesn't seem useful from any technical standpoint; I think that is purely ideological. In any case, it's all a bit imperfect anyway, since it's from the perspective of the package manager, which can't be absolutely sure there's no blobs. Anyone who follows Linux-libre releases can see how hard it really is to find all of those needles in the haystack. (And yeah, it would be fantastic if we could have machines with zero unfree code and no blobs, but the majority of computers sold today can't meaningfully operate like that.) I actually believe there's plenty of value in the builds still being reproducible even when blobs are present: you can still verify that the supply chain is not compromised outside of the blobs. For practical reasons, most users will need to stick to limiting the amount of blobs rather than fully eliminating them. [1]: https://nixos.org/manual/nixpkgs/stable/#sec-meta-license https://nixos.org/manual/nixpkgs/stable/#sec-meta-license [2]: https://nixos.org/manual/nixpkgs/stable/#sec-meta-sourceProvenance https://nixos.org/manual/nixpkgs/stable/#sec-meta-sourceProv...
- 0x69420 2y ago
  you can slap a hash on a binary distribution and it becomes "reproducible" in the same trivial sense as any source tarball. after that, the reproducibility of whatever "build process" takes place to extract archives and shuffle assets around is no more or less fraught than any other package (probably less considering how much compilers have historically had to be brought to heel, especially before reproducibility was fashionable enough for it to enter much into compiler authors' consideration!!)
colordrops 2y ago
I'm curious, why couldn't packages that are fully reproduceable be marked with metadata, and in your config you set a flag to only allow reproduceable packages? Similar to the nonfree tag. Then you'd have a 100% reproduceable OS if you have the flag set (assuming that required base packages are reproduceable)
- jchw 2y ago
  You could definitely do that, I think the main thing stopping anyone is simply lack of demand for that specific feature. That, and also it might be hard to keep track of what things are properly reproducible; you can kind of only ever prove for sure that a package is not reproducible. It could be non-deterministic but only produce differences on different CPUs or an infinitesimally small percentage of times. Actually being able to assure determinism would be pretty amazing although I don't know how that could be achieved.
  - colordrops 2y ago
    I assume it would be somewhat of a judgement call. I mean that is the case with nonfree packages as well - licenses and whatnot have to be evaluated. I assume that there are no cases of non-trivially large software packages in the wild that have been formally proven to be reproducible, but I could be wrong.
k__ 2y ago
What are the common issues besides timestamps?
- colejohnson66 2y ago
  Compiler executing internal work concurrently and merging at the end. Thread scheduling changes will cause a different output ordering.

by jchw 2y ago

sa46 2y ago

> It is closer to being fully reproducible than most other build systems (including Bazel). How so? Bazel produces the same results for the same inputs.

jchw 2y ago
Bazel doesn't guarantee bit-exact outputs, but also Bazel doesn't guarantee pure builds. It does have a sandbox that prevents some impurities, but for example it doesn't prevent things from going out to the network, or even accessing files from anywhere in the filesystem, if you use absolute paths. (Although, on Linux at least, Bazel does prevent you from modifying files outside of the sandbox directory.) The Nix sandbox does completely obscure the host filesystem and limit network access to processes that can produce a bit-exact output only. (Bazel also obviously uses the system compilers and headers. Nix does not.)
- dijit 2y ago
  Uh, Either my understanding of Bazel is wrong, or everything you wrote is wrong. Bazel absolutely prevents network access and filesystem access (reads) from builds. (only permitting explicit network includes from the WORKSPACE file, and access to files explicitly depended on in the BUILD files). Maybe you can write some “rules_” for languages that violate this, but it is designed purposely to be hermetic and bit-perfect reproducible. EDIT: From the FAQ[0]: > Will Bazel make my builds reproducible automatically? > For Java and C++ binaries, yes, assuming you do not change the toolchain. The issues with Docker's style of "reproducible" (meaning.. consistent environment; are also outlined in the same FAQ[1] > Doesn’t Docker solve the reproducibility problems? > Docker does not address reproducibility with regard to changes in the source code. Running Make with an imperfectly written Makefile inside a Docker container can still yield unpredictable results. [0]: https://bazel.build/about/faq#will_bazel_make_my_builds_reproducible_automatically https://bazel.build/about/faq#will_bazel_make_my_builds_repr... [1]: https://bazel.build/about/faq#doesn’t_docker_solve_the_reproducibility_problems https://bazel.build/about/faq#doesn’t_docker_solve_the_repro...
  - valcron1000 2y ago
    I'm not familiar with Bazel at all so this might be obvious, but does Bazel check that the files listed in the BUILD file are the "right ones" (ex. through a checksum), and if so, is this always enforced (that is, this behavior cannot be disabled)?
    - dijit 2y ago
      The contents of files are basically hashed, if the contents don't change of the file listed for a target then no change will happen, even if you modify metadata of the file (like last modified time by `touch` or so on.) Bazel is really sophisticated and I'd be lying if I said I understood it well, but I have spent time looking at it.
  - keithwinstein 2y ago
    I think you're both right in a sense. Bazel doesn't (in general) prevent filesystem access, e.g. to library headers in /usr/include. If those headers change (maybe because a Debian package got upgraded or whatever), Bazel won't know it has to invalidate the build cache. I think the FAQ is still technically correct because upgrading the Debian package for a random library dependency counts as "chang[ing] the toolchain" in this context. But I don't think you'd call it hermetic by default. Check out the previous discussion at https://news.ycombinator.com/item?id=23184843 https://news.ycombinator.com/item?id=23184843 and below: > Under the hood there's a default auto-configured toolchain that finds whatever is installed locally in the system. Since it has no way of knowing what files an arbitrary "cc" might depend on, you lose hermeticity by using it.
gf000 2y ago
No, most compilers are not themselves reproducible, even within very restrictive sandboxes (e.g. they may do some work concurrently and collect the results based on when it completes, then build on top of that. If they don't add a timing-insensitive sorting step, the resulting binary will (assuming no bugs) be functionally equivalent, but may not be bit-by-by equal), and a build tool can only do so much.

matrss 2y ago

> The quest to get every build process to be deterministic [...] will never be solved for all of Nixpkgs. Not least because of unfree and/or binary-blob packages that can't be reproducible because they don't even build anything. As much as Guix' strict FOSS and build-from-source policy can be an annoyance, it is a necessary precondition to achieve full reproducibility from source, i.e. the full-source bootstrap.

jchw 2y ago
Nixpkgs provides license[1] and source provenance[2] information. For legal reasons, Nix also defaults to not evaluating unfree packages. Not packaging them at all, though, doesn't seem useful from any technical standpoint; I think that is purely ideological. In any case, it's all a bit imperfect anyway, since it's from the perspective of the package manager, which can't be absolutely sure there's no blobs. Anyone who follows Linux-libre releases can see how hard it really is to find all of those needles in the haystack. (And yeah, it would be fantastic if we could have machines with zero unfree code and no blobs, but the majority of computers sold today can't meaningfully operate like that.) I actually believe there's plenty of value in the builds still being reproducible even when blobs are present: you can still verify that the supply chain is not compromised outside of the blobs. For practical reasons, most users will need to stick to limiting the amount of blobs rather than fully eliminating them. [1]: https://nixos.org/manual/nixpkgs/stable/#sec-meta-license https://nixos.org/manual/nixpkgs/stable/#sec-meta-license [2]: https://nixos.org/manual/nixpkgs/stable/#sec-meta-sourceProvenance https://nixos.org/manual/nixpkgs/stable/#sec-meta-sourceProv...
0x69420 2y ago
you can slap a hash on a binary distribution and it becomes "reproducible" in the same trivial sense as any source tarball. after that, the reproducibility of whatever "build process" takes place to extract archives and shuffle assets around is no more or less fraught than any other package (probably less considering how much compilers have historically had to be brought to heel, especially before reproducibility was fashionable enough for it to enter much into compiler authors' consideration!!)

colordrops 2y ago

I'm curious, why couldn't packages that are fully reproduceable be marked with metadata, and in your config you set a flag to only allow reproduceable packages? Similar to the nonfree tag. Then you'd have a 100% reproduceable OS if you have the flag set (assuming that required base packages are reproduceable)

jchw 2y ago
You could definitely do that, I think the main thing stopping anyone is simply lack of demand for that specific feature. That, and also it might be hard to keep track of what things are properly reproducible; you can kind of only ever prove for sure that a package is not reproducible. It could be non-deterministic but only produce differences on different CPUs or an infinitesimally small percentage of times. Actually being able to assure determinism would be pretty amazing although I don't know how that could be achieved.
- colordrops 2y ago
  I assume it would be somewhat of a judgement call. I mean that is the case with nonfree packages as well - licenses and whatnot have to be evaluated. I assume that there are no cases of non-trivially large software packages in the wild that have been formally proven to be reproducible, but I could be wrong.

k__ 2y ago

What are the common issues besides timestamps?

colejohnson66 2y ago
Compiler executing internal work concurrently and merging at the end. Thread scheduling changes will cause a different output ordering.