4 ms·

Can anyone comment on the significance of this accomplishment, and why it was hard to achieve before? I (naively, apparently) assumed this had been possible wi

by avalys 5y ago

Can anyone comment on the significance of this accomplishment, and why it was hard to achieve before? I (naively, apparently) assumed this had been possible with open-source toolchains for a long time.

xyzzy_plugh 5y ago
There's a lot of problems with reproducible builds. Filesystem paths, timestamps, deterministic build order to say the least. This is a pretty great achievement and I'm looking forward to a non-minimal stable ISO.
- bombcar 5y ago
  Yeah even the “gcc compiled Jan 23, 2021 at 11:23AM” messages you often see breaks deterministic builds.
twisrkrr 5y ago
The code has to be changed so that things like system specific paths, time of compilation, hardware, etc. Don’t cause the compiled program to be unique to that computer (meaning compiling the same code on a different computer will give you a file that still works but has a different md5 hash) By being able to reproduce the file completely, down to identical md5 hashes, you know you have the same file the creator has, and know with certainty that the file has not been tampered with
- secondcoming 5y ago
  Does this mean that the code cannot be built with CPU specific optimisations (march option with gcc)
  - Avamander 5y ago
    Pretty much. But hopefully x86_64 feature levels will provide the benefits of native builds to a reasonable extent.
  - Denvercoder9 5y ago
    The software doesn't suddenly become incompatible with CPU-specific optimisations (or many other compiler flags that change its output), but if you do so, you won't be able to reproduce the distribution binaries. Distributions don't enable CPU-specific optimisations anyway, since they want to be usable on more than one CPU model.
  - pas 5y ago
    Likely it means that with the same input arguments the end result is bit-by-bit identical. (As I understand the problems were hard to control output elements. So it was not enough to se the same args, set the same time, and use the same path and filesystem, because there were things that happened at different speeds, so they ended up happening at relative different elapsed times, so the outputs contained different timestamps, etc.)
  - clhodapp 5y ago
    No, just that you need to avoid naively conflating the machine that is doing the compilation with the one that optimization is being performed for. Concretely, you would need to keep track of and reproduce e.g. the march flag value as a part of your build input. If you wanted to optimize for multiple architectures, that would mean separate builds or a larger binary with function multi-versioning.
  - maartenh 5y ago
    Nixpkgs contains the build / patch instructions for any packages in NixOS. If you want to compile any piece of software available in Nixpkgs, you can override it's attributes (inputs used to build it). One can trivially have an almost identical operation system to your colleagues install, but override just one package to enable optimisations for a certain cpu. This would however imply that you'd lose the transparent binary cache that you could otherwise use. Exactly this method is used to configure the entire operating install! Your OS install is just another package that has some custom inputs set.
danbst 5y ago
Just recently, there were large non-reproducible projects: python, gcc. Not sure where is the history of non-r13y. --- There is Debian initiative to create bit-to-bit reproducible builds for all their software (well, all critical). https://reproducible-builds.org/ https://reproducible-builds.org/ R13y is akin to "computer proofs" in math -- if you don't have it, that's fine, but if you have it, that's awesome. There are practical reasons to favor reproducibility too, but those are more for distro maintainers. The fact that NixOS (not Debian) got this 100% is mostly because - minimal image has a small subset of packages (https://hydra.nixos.org/build/146009592#tabs-build-deps https://hydra.nixos.org/build/146009592#tabs-build-deps) - Nix tooling was created 15 years ago *exactly* for this, Nix is mad to make packages bit-to-bit rebuildable from scratch. - Nix/Nixpkgs is growing in number of maintainers and got more funds - Nix has fewer Docker/Snap pragmatics
- infogulch 5y ago
  Being bit-for-bit reproduceable means you could do fun things like distribute packages as just sources and a big blob of signatures, and you can still run only signed binaries.
- Foxboron 5y ago
  >- Nix tooling was created 15 years ago exactly for this, Nix is mad to make packages bit-to-bit rebuildable from scratch. I don't think this is accurate? Nix is about reproducing system behaviour, largely by capturing the dependency graph and replaying the build. But this doesn't entail bit-for-bit identical binaries. It's very much sits in the same group such as Docker and similar technologies. This is also how I read the original thesis from Eelco[0]. And well, claims like this always rubs me the wrong way since nixos only really started using the word "reproducible builds" after Debian started their efforts in 2015-2016[1], and started their reproducible builds effort later. It also muddies the language since people are now talking about "reproducible builds" in terms of system behavior as well as bit-for-bit identical builds. The result has been that people talk about "verifiable builds" instead. [0]: https://edolstra.github.io/pubs/phd-thesis.pdf https://edolstra.github.io/pubs/phd-thesis.pdf [1]: https://github.com/NixOS/nixpkgs/issues/9731 https://github.com/NixOS/nixpkgs/issues/9731
- 5y ago
mananaysiempre 5y ago
The GCC developers in particular were hostile to such efforts for a long time, IIRC. (This is a non-trivial issue because randomized data structures exist and can be a good idea to use: treaps, universal hashes, etc. I’d guess it also pays for compiler heuristics to be randomized sometimes. Incremental compilation is much harder to achieve when you require bit-for-bit identical output. Even just stripping your compile paths from debug info is not entirely straightforward.)
- pas 5y ago
  How/why was the randomness part not "solveable" via using fixed seeds?
  - bruce343434 5y ago
    the security benefit of things like stack canaries rest on them being random and not known beforehand, I guess. Otherwise stack smashing malware could know to avoid them.
    - mananaysiempre 5y ago
      Wait, how is that relevant? Nothing says stack canaries have to use the same RNG as the main program, let alone the same seed, and there are cases such as this one where they probably shouldn’t, so it makes sense to separate them.
- moonchild 5y ago
  > Incremental compilation is much harder to achieve when you require bit-for-bit identical output Presumably, incremental compilation is only for development. For release, you would do a clean build, which would be reproducible. > Even just stripping your compile paths from debug info is not entirely straightforward Just use the same paths.
  - mananaysiempre 5y ago
    > Presumably, incremental compilation is only for development. For release, you would do a clean build, which would be reproducible. I’d say that’s exactly the wrong approach: given how hard incremental anything is, it would make sense to insist on bit-exact output and then fuzz the everliving crap out of it until bit-exactness was reached. (The GCC maintainers do not agree.) But yes, you could do that. It’s not impossible to do reproducible builds with GCC 4.7 or whatever, it’s just intensely unpleasant, especially as a distro maintainer faced with yet another bespoke build system. (Saying that with all the self-awareness of a person making their own build system.) > Just use the same paths. I mean, sure, but then you have to build and debug in a chroot and waste half a day of your life figuring out how to do that and just generally feel stupid. And your debug info is still useless to anybody not using the exact same setup. Can’t we just embed relative paths instead, or even arbitrary prefixes it the code is coming from more than one place? In recent GCC versions we can, just chuck the right incantation into CPPFLAGS and you’re golden. All of this is not really difficult except insofar as getting a large and complicated program to do anything is difficult. (Stares in the direction of the 17-year-old Firefox bug for XDG basedir support.) That’s why I said it wasn’t a GCC problem so much as a maintainer attitude problem.
deleted 5y ago
[deleted]
peterkelly 5y ago
For some reason, many compilers and build scripts have traditionally been written in a way that's not referentially transparent (a pure function from input to output). Unnecessary information like the time of the build, absolute path names of sources and intermediate files, usernames and hostnames often would find their way into build outputs. Compiling the same source on different machines or at different times would yield different results. Reproducible builds avoid all this and always produce the same outputs given the same inputs. There's no good reason (that I can think of) why this shouldn't have been the case all along, but for a long time I guess it just wasn't seen as a priority. The benefit of reproducible builds is that it's possible to verify that a distributed binary was definitely compiled from known source files and hasn't been tampered with, because you can recompile the program yourself and check that the result matches the binary distribution.
- dane-pgp 5y ago
  > There's no good reason (that I can think of) why this shouldn't have been the case all along Well, it's not like developers consciously thought "How can I make my build process as non-deterministic as possible?", it's just that by the time people started to become aware of the benefits of reproducibility, various forms of non-determinism had already crept in. For example, someone writing an archiving tool would be completely right to think it is a useful feature to store the creation date of the archive in the archive's metadata. The idea that a user might want to force this value to instead be some fixed constant would only occur to someone later when they noticed that their packages were non-reproducible because of this. But you're right; if the goal had been thought of from the start, there's no reason why every build tool wouldn't have supported this.
  - russfink 5y ago
    Thank you both. I was wondering the same thing.
- otabdeveloper4 5y ago
  > The benefit of reproducible builds is that it's possible to verify that a distributed binary was definitely compiled from known source files and hasn't been tampered with, because you can recompile the program yourself and check that the result matches the binary distribution. It's not just security. If a hash of the input sources maps directly to a hash of the output binaries, then you can automatically cache build artefacts by hash tag and get huge speedups when compiling stuff from scratch. This was the primary motivation for Nix, since Nix does a whole lot of building from scratch and caching.

by avalys 5y ago

xyzzy_plugh 5y ago

There's a lot of problems with reproducible builds. Filesystem paths, timestamps, deterministic build order to say the least. This is a pretty great achievement and I'm looking forward to a non-minimal stable ISO.

bombcar 5y ago
Yeah even the “gcc compiled Jan 23, 2021 at 11:23AM” messages you often see breaks deterministic builds.

twisrkrr 5y ago

The code has to be changed so that things like system specific paths, time of compilation, hardware, etc. Don’t cause the compiled program to be unique to that computer (meaning compiling the same code on a different computer will give you a file that still works but has a different md5 hash) By being able to reproduce the file completely, down to identical md5 hashes, you know you have the same file the creator has, and know with certainty that the file has not been tampered with

secondcoming 5y ago
Does this mean that the code cannot be built with CPU specific optimisations (march option with gcc)
- Avamander 5y ago
  Pretty much. But hopefully x86_64 feature levels will provide the benefits of native builds to a reasonable extent.
- Denvercoder9 5y ago
  The software doesn't suddenly become incompatible with CPU-specific optimisations (or many other compiler flags that change its output), but if you do so, you won't be able to reproduce the distribution binaries. Distributions don't enable CPU-specific optimisations anyway, since they want to be usable on more than one CPU model.
- pas 5y ago
  Likely it means that with the same input arguments the end result is bit-by-bit identical. (As I understand the problems were hard to control output elements. So it was not enough to se the same args, set the same time, and use the same path and filesystem, because there were things that happened at different speeds, so they ended up happening at relative different elapsed times, so the outputs contained different timestamps, etc.)
- clhodapp 5y ago
  No, just that you need to avoid naively conflating the machine that is doing the compilation with the one that optimization is being performed for. Concretely, you would need to keep track of and reproduce e.g. the march flag value as a part of your build input. If you wanted to optimize for multiple architectures, that would mean separate builds or a larger binary with function multi-versioning.
- maartenh 5y ago
  Nixpkgs contains the build / patch instructions for any packages in NixOS. If you want to compile any piece of software available in Nixpkgs, you can override it's attributes (inputs used to build it). One can trivially have an almost identical operation system to your colleagues install, but override just one package to enable optimisations for a certain cpu. This would however imply that you'd lose the transparent binary cache that you could otherwise use. Exactly this method is used to configure the entire operating install! Your OS install is just another package that has some custom inputs set.

danbst 5y ago

Just recently, there were large non-reproducible projects: python, gcc. Not sure where is the history of non-r13y. --- There is Debian initiative to create bit-to-bit reproducible builds for all their software (well, all critical). https://reproducible-builds.org/ https://reproducible-builds.org/ R13y is akin to "computer proofs" in math -- if you don't have it, that's fine, but if you have it, that's awesome. There are practical reasons to favor reproducibility too, but those are more for distro maintainers. The fact that NixOS (not Debian) got this 100% is mostly because - minimal image has a small subset of packages (https://hydra.nixos.org/build/146009592#tabs-build-deps https://hydra.nixos.org/build/146009592#tabs-build-deps) - Nix tooling was created 15 years ago *exactly* for this, Nix is mad to make packages bit-to-bit rebuildable from scratch. - Nix/Nixpkgs is growing in number of maintainers and got more funds - Nix has fewer Docker/Snap pragmatics

infogulch 5y ago
Being bit-for-bit reproduceable means you could do fun things like distribute packages as just sources and a big blob of signatures, and you can still run only signed binaries.
Foxboron 5y ago
>- Nix tooling was created 15 years ago exactly for this, Nix is mad to make packages bit-to-bit rebuildable from scratch. I don't think this is accurate? Nix is about reproducing system behaviour, largely by capturing the dependency graph and replaying the build. But this doesn't entail bit-for-bit identical binaries. It's very much sits in the same group such as Docker and similar technologies. This is also how I read the original thesis from Eelco[0]. And well, claims like this always rubs me the wrong way since nixos only really started using the word "reproducible builds" after Debian started their efforts in 2015-2016[1], and started their reproducible builds effort later. It also muddies the language since people are now talking about "reproducible builds" in terms of system behavior as well as bit-for-bit identical builds. The result has been that people talk about "verifiable builds" instead. [0]: https://edolstra.github.io/pubs/phd-thesis.pdf https://edolstra.github.io/pubs/phd-thesis.pdf [1]: https://github.com/NixOS/nixpkgs/issues/9731 https://github.com/NixOS/nixpkgs/issues/9731
5y ago

mananaysiempre 5y ago

The GCC developers in particular were hostile to such efforts for a long time, IIRC. (This is a non-trivial issue because randomized data structures exist and can be a good idea to use: treaps, universal hashes, etc. I’d guess it also pays for compiler heuristics to be randomized sometimes. Incremental compilation is much harder to achieve when you require bit-for-bit identical output. Even just stripping your compile paths from debug info is not entirely straightforward.)

pas 5y ago
How/why was the randomness part not "solveable" via using fixed seeds?
- bruce343434 5y ago
  the security benefit of things like stack canaries rest on them being random and not known beforehand, I guess. Otherwise stack smashing malware could know to avoid them.
  - mananaysiempre 5y ago
    Wait, how is that relevant? Nothing says stack canaries have to use the same RNG as the main program, let alone the same seed, and there are cases such as this one where they probably shouldn’t, so it makes sense to separate them.
moonchild 5y ago
> Incremental compilation is much harder to achieve when you require bit-for-bit identical output Presumably, incremental compilation is only for development. For release, you would do a clean build, which would be reproducible. > Even just stripping your compile paths from debug info is not entirely straightforward Just use the same paths.
- mananaysiempre 5y ago
  > Presumably, incremental compilation is only for development. For release, you would do a clean build, which would be reproducible. I’d say that’s exactly the wrong approach: given how hard incremental anything is, it would make sense to insist on bit-exact output and then fuzz the everliving crap out of it until bit-exactness was reached. (The GCC maintainers do not agree.) But yes, you could do that. It’s not impossible to do reproducible builds with GCC 4.7 or whatever, it’s just intensely unpleasant, especially as a distro maintainer faced with yet another bespoke build system. (Saying that with all the self-awareness of a person making their own build system.) > Just use the same paths. I mean, sure, but then you have to build and debug in a chroot and waste half a day of your life figuring out how to do that and just generally feel stupid. And your debug info is still useless to anybody not using the exact same setup. Can’t we just embed relative paths instead, or even arbitrary prefixes it the code is coming from more than one place? In recent GCC versions we can, just chuck the right incantation into CPPFLAGS and you’re golden. All of this is not really difficult except insofar as getting a large and complicated program to do anything is difficult. (Stares in the direction of the 17-year-old Firefox bug for XDG basedir support.) That’s why I said it wasn’t a GCC problem so much as a maintainer attitude problem.

deleted 5y ago

[deleted]

peterkelly 5y ago

For some reason, many compilers and build scripts have traditionally been written in a way that's not referentially transparent (a pure function from input to output). Unnecessary information like the time of the build, absolute path names of sources and intermediate files, usernames and hostnames often would find their way into build outputs. Compiling the same source on different machines or at different times would yield different results. Reproducible builds avoid all this and always produce the same outputs given the same inputs. There's no good reason (that I can think of) why this shouldn't have been the case all along, but for a long time I guess it just wasn't seen as a priority. The benefit of reproducible builds is that it's possible to verify that a distributed binary was definitely compiled from known source files and hasn't been tampered with, because you can recompile the program yourself and check that the result matches the binary distribution.

dane-pgp 5y ago
> There's no good reason (that I can think of) why this shouldn't have been the case all along Well, it's not like developers consciously thought "How can I make my build process as non-deterministic as possible?", it's just that by the time people started to become aware of the benefits of reproducibility, various forms of non-determinism had already crept in. For example, someone writing an archiving tool would be completely right to think it is a useful feature to store the creation date of the archive in the archive's metadata. The idea that a user might want to force this value to instead be some fixed constant would only occur to someone later when they noticed that their packages were non-reproducible because of this. But you're right; if the goal had been thought of from the start, there's no reason why every build tool wouldn't have supported this.
- russfink 5y ago
  Thank you both. I was wondering the same thing.
otabdeveloper4 5y ago
> The benefit of reproducible builds is that it's possible to verify that a distributed binary was definitely compiled from known source files and hasn't been tampered with, because you can recompile the program yourself and check that the result matches the binary distribution. It's not just security. If a hash of the input sources maps directly to a hash of the output binaries, then you can automatically cache build artefacts by hash tag and get huge speedups when compiling stuff from scratch. This was the primary motivation for Nix, since Nix does a whole lot of building from scratch and caching.