6 ms·

Towards a fully reproducible Debian

145 points by lamby 8y ago

nikivi 8y ago
I thought NixOS solves the full reproducibility problem really well already. https://nixos.org https://nixos.org
- Foxboron 8y ago
  It doesn't. NixOS doesn't do reproducible builds, it does reproducible systems. Reproducible builds is essentially about compiling source code to a binary and always get the same checksum Reproducible systems is about always getting the same system/behavior given the same configuration. Which is what NixOS does.
  - nikivi 8y ago
    I thought that getting the same system/behavior would require reproducible builds too. Perhaps I am wrong.
    - vertex-four 8y ago
      The same behaviour can come out of different binaries - the issue that reproducible builds is solving isn’t that programs have different behaviour from compile to compile, it’s that people can’t prove they were compiled correctly.
    - black_puppydog 8y ago
      If your build process always produces the exact same binaries, and you always use the exact same environment to run these, then yes, that's reproducible. It's not equivalent though, in that you can easily have a system that always exhibits the same behaviour (what NixOS does) but does not use bit-by-bit equal binaries. For example, some symbols in some library's .so file might be in a different order, or some padding bits that are never read might be zero or random depending on your compiler's choice.
  - aseipp 8y ago
    We did have a bunch of patches to actually make a lot of builds reproducible, but integrating changes like that has been pretty slow. I'm pretty sure a lot of these patches have since been adopted, pulling things much closer to bit-by-bit reproducible. Unfortunately there's no real metric we have to keep track of that at the moment, like Debian ("%XY.Z are working"). But it looks like basically all distros are finally starting to converge on this.
    - Foxboron 8y ago
      There was no intentions to claim you where not working towards reproducible builds. I have come across several people that conflate what NixOS offers, and what reproducible builds are. However, great that you have been working towards this! Arch Linux has been cooperating with Debian and currently build our packages on their infrastructure to produce graphs and test our packages. I assume NixOS can do the same thing if the manpower is present. https://tests.reproducible-builds.org/archlinux/archlinux.html https://tests.reproducible-builds.org/archlinux/archlinux.ht...
- yc-kraln 8y ago
  Did you read the article? It's about Debian, and it explicitly mentions NixOS. It also mentions certain packages (Emacs, certain secure boot stuff) which at the moment are not reproducible.
- chriswarbo 8y ago
  Nix doesn't "solve" reproducibility. For example, if the thing being packaged builds non-deterministically (e.g. if components are built concurrently, and added to a tarball in whichever order they finish). Nix can make it easier to avoid introducing new problems, and it also gives an audit trail of which versions of things were used to build things. Also Debian doesn't use Nix, so that wouldn't solve the problem for Debian. Nix (along with everyone else) does benefit from the improvements that Debian is making, e.g. finding non-determinism in programs and getting it fixed upstream.
algorithm314 8y ago
NetBSD has fully reproducible builds https://blog.netbsd.org/tnf/entry/netbsd_fully_reproducible_builds https://blog.netbsd.org/tnf/entry/netbsd_fully_reproducible_...
- foobarbazetc 8y ago
  The scope here is much larger as it’s every package in the Debian distro, not just the base system.
  - cat199 8y ago
    and pkgsrc has bulk package builds. same pkgsrc checkout + reproducable base => essentially reproducable full package builds. yes, some people will find some niggling way to make this not true because the binaries may be somehow altered due to timestamps, yadda, but for all practical purposes w/r/t the actual code and actual executables generated, this is true, and has been true for essentially the entire existance of bsd derivitaves using ports systems (1996).
    - lomnakkus 8y ago
      > same pkgsrc checkout + reproducable base => essentially reproducable full package builds. I would be truly surpised if that were the case. Are there no packages in pkgsrc which embed, e.g. timestamps, or even download things while building[1]? [1] JDK packages are a typical culprit in this type of situation because Oracle JDK requires an "accept license" prompt. > es, some people will find some niggling way to make this not true because the binaries may be somehow altered due to timestamps, yadda, but for all practical purposes w/r/t the actual code and actual executables generated, this is true, and has been true for essentially the entire existance of bsd derivitaves using ports systems (1996). Oh, so you're coopting "essentially reproducable" to mean "not reproducible". Ok then. "Reproducible builds" is about verification and isn't just about "close enough".
    - foobarbazetc 8y ago
      Did you... read the article?
geokon 8y ago
So I'm glad this is being done, but the "attack surface" that's being resolved sounds like a bit of a joke... "Alice, a system administrator who contributes to a Linux distribution, is building her binaries on servers that, unknown to her, have been compromised" This is a bit of weird scenario.. so they're assuming the only build being compromised is the one that ends up in the repo and no one can confirm that easily. So just have two (or more) identical servers in separate locations under different people's control so they aren't both compromised? No need to fuzz the dates and paths and stuff. Or if you really want, you make a special Debian version (super-stable) with fixed clocks, file paths, etc. where things are easy to reproduce. "Bob, a privacy-oriented developer, makes a privacy-preserving browser, but is being blackmailed into secretly including vulnerabilities in the binaries he provides. Carol is a free-software user whose laptop is being attacked by an evil maid called Eve, the third developer of the title; each time Carol shares free software with her friends, it is pre-compromised by Eve" Do Debian contributors just upload a binary blob to the Debian servers and that's it? I though they were saying they have build servers for every package and the source from which it's built is available How about even easier - a bribed/black-mailed Debian maintainer slips in a patch to some of the tens of thousands of packages they tweak, no one notices and it gets distributed to everyone. I'm not expert, but it seems the smarter solution would be to have a much smaller reproducible core/base system (with Debian patches when necessary). Something that can realistically get enough eyeballs on it. And then everything else is built by project maintainers into something like flatpaks/snaps with no patching.
- geezerjay 8y ago
  > So just have two (or more) identical servers in separate locations under different people's control so they aren't both compromised? Considering that you are a DNS spoof away from installing packages from a compromised server, how does your proposal tackle the underlying problem?
  - danieldk 8y ago
    Considering that you are a DNS spoof away from installing packages from a compromised server, Virtually all package managers now verify package and/or metadata signatures. If a server is spoofed and the system downloads a compromised package, the package manager will simply refuse to install it, because it is not signed by your distribution's packages. Reproducible builds are useful because it reduces the probability of a compromised build system going undetected, by building the package on several independent builders. For the user it also answers the question: does a distributor actually compile from the source code that they claim to compile from?
verytrivial 8y ago
I would gladly set a "preproducible_only=yes" or whatever config option to exclude non-reproducible packages from a personal system. This would create a strong incentive to contribute patches. (I'm hoping someone can point out such a setting already exists! It didn't last time I checked.)
- rkangel 8y ago
  Unfortunately, such a flag would probably result in weird package install/update errors as part of the dependency graph would be missing.
  - BenjiWiebe 8y ago
    Wouldn't have to be a wierd error message. "The package foo cannot be installed because it's dependency bar cannot be reproducibly built."
  - rcthompson 8y ago
    Can a package even be considered reproducible if its dependencies aren't?
    - verytrivial 8y ago
      I would argue not, but that's the point, right? Do we have a large enough well-connected graph of reproducibly buildable packages, and if not, what should be fixed first?
    - lozenge 8y ago
      That depends on whether the dependencies are build dependencies.
- lamby 8y ago
  So, this actually exists as an extremely WIP patch: https://bugs.debian.org/863622 https://bugs.debian.org/863622 Underling that you should not apply this patch and/or it is simply a demonstration of one potential user-interface.

145 points by lamby 8y ago

nikivi 8y ago

I thought NixOS solves the full reproducibility problem really well already. https://nixos.org https://nixos.org

Foxboron 8y ago
It doesn't. NixOS doesn't do reproducible builds, it does reproducible systems. Reproducible builds is essentially about compiling source code to a binary and always get the same checksum Reproducible systems is about always getting the same system/behavior given the same configuration. Which is what NixOS does.
- nikivi 8y ago
  I thought that getting the same system/behavior would require reproducible builds too. Perhaps I am wrong.
  - vertex-four 8y ago
    The same behaviour can come out of different binaries - the issue that reproducible builds is solving isn’t that programs have different behaviour from compile to compile, it’s that people can’t prove they were compiled correctly.
  - black_puppydog 8y ago
    If your build process always produces the exact same binaries, and you always use the exact same environment to run these, then yes, that's reproducible. It's not equivalent though, in that you can easily have a system that always exhibits the same behaviour (what NixOS does) but does not use bit-by-bit equal binaries. For example, some symbols in some library's .so file might be in a different order, or some padding bits that are never read might be zero or random depending on your compiler's choice.
- aseipp 8y ago
  We did have a bunch of patches to actually make a lot of builds reproducible, but integrating changes like that has been pretty slow. I'm pretty sure a lot of these patches have since been adopted, pulling things much closer to bit-by-bit reproducible. Unfortunately there's no real metric we have to keep track of that at the moment, like Debian ("%XY.Z are working"). But it looks like basically all distros are finally starting to converge on this.
  - Foxboron 8y ago
    There was no intentions to claim you where not working towards reproducible builds. I have come across several people that conflate what NixOS offers, and what reproducible builds are. However, great that you have been working towards this! Arch Linux has been cooperating with Debian and currently build our packages on their infrastructure to produce graphs and test our packages. I assume NixOS can do the same thing if the manpower is present. https://tests.reproducible-builds.org/archlinux/archlinux.html https://tests.reproducible-builds.org/archlinux/archlinux.ht...
yc-kraln 8y ago
Did you read the article? It's about Debian, and it explicitly mentions NixOS. It also mentions certain packages (Emacs, certain secure boot stuff) which at the moment are not reproducible.
chriswarbo 8y ago
Nix doesn't "solve" reproducibility. For example, if the thing being packaged builds non-deterministically (e.g. if components are built concurrently, and added to a tarball in whichever order they finish). Nix can make it easier to avoid introducing new problems, and it also gives an audit trail of which versions of things were used to build things. Also Debian doesn't use Nix, so that wouldn't solve the problem for Debian. Nix (along with everyone else) does benefit from the improvements that Debian is making, e.g. finding non-determinism in programs and getting it fixed upstream.

algorithm314 8y ago

NetBSD has fully reproducible builds https://blog.netbsd.org/tnf/entry/netbsd_fully_reproducible_builds https://blog.netbsd.org/tnf/entry/netbsd_fully_reproducible_...

foobarbazetc 8y ago
The scope here is much larger as it’s every package in the Debian distro, not just the base system.
- cat199 8y ago
  and pkgsrc has bulk package builds. same pkgsrc checkout + reproducable base => essentially reproducable full package builds. yes, some people will find some niggling way to make this not true because the binaries may be somehow altered due to timestamps, yadda, but for all practical purposes w/r/t the actual code and actual executables generated, this is true, and has been true for essentially the entire existance of bsd derivitaves using ports systems (1996).
  - lomnakkus 8y ago
    > same pkgsrc checkout + reproducable base => essentially reproducable full package builds. I would be truly surpised if that were the case. Are there no packages in pkgsrc which embed, e.g. timestamps, or even download things while building[1]? [1] JDK packages are a typical culprit in this type of situation because Oracle JDK requires an "accept license" prompt. > es, some people will find some niggling way to make this not true because the binaries may be somehow altered due to timestamps, yadda, but for all practical purposes w/r/t the actual code and actual executables generated, this is true, and has been true for essentially the entire existance of bsd derivitaves using ports systems (1996). Oh, so you're coopting "essentially reproducable" to mean "not reproducible". Ok then. "Reproducible builds" is about verification and isn't just about "close enough".
  - foobarbazetc 8y ago
    Did you... read the article?

geokon 8y ago

So I'm glad this is being done, but the "attack surface" that's being resolved sounds like a bit of a joke... "Alice, a system administrator who contributes to a Linux distribution, is building her binaries on servers that, unknown to her, have been compromised" This is a bit of weird scenario.. so they're assuming the only build being compromised is the one that ends up in the repo and no one can confirm that easily. So just have two (or more) identical servers in separate locations under different people's control so they aren't both compromised? No need to fuzz the dates and paths and stuff. Or if you really want, you make a special Debian version (super-stable) with fixed clocks, file paths, etc. where things are easy to reproduce. "Bob, a privacy-oriented developer, makes a privacy-preserving browser, but is being blackmailed into secretly including vulnerabilities in the binaries he provides. Carol is a free-software user whose laptop is being attacked by an evil maid called Eve, the third developer of the title; each time Carol shares free software with her friends, it is pre-compromised by Eve" Do Debian contributors just upload a binary blob to the Debian servers and that's it? I though they were saying they have build servers for every package and the source from which it's built is available How about even easier - a bribed/black-mailed Debian maintainer slips in a patch to some of the tens of thousands of packages they tweak, no one notices and it gets distributed to everyone. I'm not expert, but it seems the smarter solution would be to have a much smaller reproducible core/base system (with Debian patches when necessary). Something that can realistically get enough eyeballs on it. And then everything else is built by project maintainers into something like flatpaks/snaps with no patching.

geezerjay 8y ago
> So just have two (or more) identical servers in separate locations under different people's control so they aren't both compromised? Considering that you are a DNS spoof away from installing packages from a compromised server, how does your proposal tackle the underlying problem?
- danieldk 8y ago
  Considering that you are a DNS spoof away from installing packages from a compromised server, Virtually all package managers now verify package and/or metadata signatures. If a server is spoofed and the system downloads a compromised package, the package manager will simply refuse to install it, because it is not signed by your distribution's packages. Reproducible builds are useful because it reduces the probability of a compromised build system going undetected, by building the package on several independent builders. For the user it also answers the question: does a distributor actually compile from the source code that they claim to compile from?

verytrivial 8y ago

I would gladly set a "preproducible_only=yes" or whatever config option to exclude non-reproducible packages from a personal system. This would create a strong incentive to contribute patches. (I'm hoping someone can point out such a setting already exists! It didn't last time I checked.)

rkangel 8y ago
Unfortunately, such a flag would probably result in weird package install/update errors as part of the dependency graph would be missing.
- BenjiWiebe 8y ago
  Wouldn't have to be a wierd error message. "The package foo cannot be installed because it's dependency bar cannot be reproducibly built."
- rcthompson 8y ago
  Can a package even be considered reproducible if its dependencies aren't?
  - verytrivial 8y ago
    I would argue not, but that's the point, right? Do we have a large enough well-connected graph of reproducibly buildable packages, and if not, what should be fixed first?
  - lozenge 8y ago
    That depends on whether the dependencies are build dependencies.
lamby 8y ago
So, this actually exists as an extremely WIP patch: https://bugs.debian.org/863622 https://bugs.debian.org/863622 Underling that you should not apply this patch and/or it is simply a demonstration of one potential user-interface.