5 ms·

This is a huge achievement for Debian and the free software world. It took a while though until this was understood. In 2007 when pointing out on debian-devel

by uecker 6mo ago

This is a huge achievement for Debian and the free software world. It took a while though until this was understood. In 2007 when pointing out on debian-devel that this is needed, I was still told what huge waste of time this would be. And indeed it took a huge amount of work by many people to get there, but it is well worth it.

PunchyHamster 6mo ago
There was no bug or attack on Debian since 2007 that reproducible packages would prevent. "Well worth it" is not correct. And it just ups the the contribution barrier to Debian higher, I already heard a lot of people complaining that contributing to Debian is hard and while in past I defended it by "they need all the checks and bounds to make sure packages play with eachother nicely", this is just step that makes it hard for no reason and little benefit.
- aborsy 6mo ago
  There was perhaps no detected bug or attack. There have most likely been bugs or attacks that reproducible builds would have prevented.
  - PunchyHamster 6mo ago
    And you base it on what exactly ? It's "just" making sure the build process is always ordered. If anything it will make attacker's job easier, as Ubuntu package will have same files structured exactly same way as Debian one.
    - yjftsjthsd-h 6mo ago
      > as Ubuntu package will have same files structured exactly same way as Debian one. As opposed to what? If Ubuntu uses the same source, of course they get the same binaries. And if Ubuntu applies patches, they'll get something different. And that's still true.
  - CyberDildonics 6mo ago
    There have most likely been bugs or attacks that reproducible builds would have prevented. Like what exactly?
- savolai 6mo ago
  ” If you are wondering why we are doing this at all, then hopefully the Reproducible Builds website will explain why this is useful.” https://reproducible-builds.org/ https://reproducible-builds.org/ Could you perhaps respond to the argumentation here?
  - dvogel 6mo ago
    (Not OP, but...) I still fail to see the current value in confirming that a reproducing builder also included the same compromised dependency that I did when I built it. I understand that reproducible builds are guarding against dynamic attacks within build infrastructure. However I just don't see those happening. Compromised source dependencies are a 100x more common problem.
    - deleted 6mo ago
      [deleted]
    - fsflover 6mo ago
      https://en.wikipedia.org/wiki/XZ_Utils_backdoor https://en.wikipedia.org/wiki/XZ_Utils_backdoor
      - mmooss 6mo ago
        https://news.ycombinator.com/item?id=48083768 https://news.ycombinator.com/item?id=48083768
      - PunchyHamster 6mo ago
        that's example of attack reproducible packages do not protect from, why you are linking it ?
        fsflover 6mo ago
        A distro automatically verifying that installed packages are reproducible would protect the user?
- azkalam 6mo ago
  Reproducible builds reduce the need for trusted parties. Have many organizations produce the binaries independently and post the arifacts. Once n of m parties agree on the arifact hash, take that as the trusted build. If every party reaches a different hash then we cannot build consensus.
  - sgc 6mo ago
    To move away from organizational dependence, there should be an installable project for debian where I can dedicate some configurable small percentage of my compute when idle to reproducibly building debian components to make a robust verification system, starting with the most critical code. Obviously, it would be a ton of work to make such a system resistant to gaming by malicious actors (see GNU Guix for useful efforts), but it would provide valuable diversity in architecture and (political or other) control. It would be even cooler if we could have independent projects that could run on various distros and OS, and build packages for any of them. Having packages for bsd verified on linux and vice-versa with statistical logging (this code has been verified x times on y OSes) would be reassuring.
    - PunchyHamster 6mo ago
      I think that project is called Ubuntu.
      - saghm 6mo ago
        Building Ubuntu does not produce identical binaries to Debian, so no, that's not what they're asking for
      - sgc 6mo ago
        I don't know of anything Ubuntu is doing that is significantly beyond what Debian is doing in this regard, nor that they have a distributed reproduction system set up???
- eptcyka 6mo ago
  It makes shipping backdoors a whole lot harder, yes.
  - zaphirplane 6mo ago
    Hmm, it prevents Trojan binaries which is a small subset of backdoor IMHO. Defense in depth obviously is a good thing
  - PunchyHamster 6mo ago
    Unless someone spins entirely separate infrastructure dedicated just for verifying Debian packages, it doesn't.
- MomsAVoxell 6mo ago
  Reproducible builds are applicable not only to respond to ‘attacks’, a subject you seem to be bikeshedding, but also for other reasons too. Anyone having to maintain a code base or a distributed fleet of devices will gain from this decision, immensely, as their operational periods come and go. Reproducible builds are about longevity as much as they are about security. Please don’t make bold claims about ‘no reason and little benefit’ while demonstrating ignorance of this hard fact: reproducible builds should have been the norm, in computing, from the get-go.
  - bluGill 6mo ago
    I longevity is harmed though. Your certs need to expire in a few years we think that your toolchain will not be downloadable.
    - MomsAVoxell 6mo ago
      Those problems need to be solved as well.
      - bluGill 6mo ago
        I don't think they do, actually. Longevity sounds good, but in reality anything that's old probably has critical security holes and so you shouldn't use it anyway.
        iamnothere 6mo ago
        A warning is sufficient. Old tech should continue to work, for preservation and archival reasons.
        PunchyHamster 6mo ago
        just archiving binary artifacts and source packages is enough Reproducibility adds nothing here
- benregenspan 6mo ago
  Is the "Jia Tan" XZ Utils compromise not a good example? That relied on code snuck into a release that was not in source. (It was caught before being promoted into a stable Debian release, yes, but this sort of relied on a happy accident, too close for comfort)
  - chuckadams 6mo ago
    The xz hack was still reproducible, because it was included in the distribution archive which did not match the upstream source -- even then, it was so obfuscated it likely would have gone unnoticed, but nevertheless it only lived in the uploaded tarball and not in the repo. Reproducibility is a good thing, but the next step is build provenance. Still, lots of good non-security benefits to reproducible builds too.
  - jcranmer 6mo ago
    The xz utils compromise is a very good example... of why reproducible packages doesn't actually solve anything security-wise! The backdoor relied first on a difference between building a package in a packaging environment versus building the package on your own. And also, it relied on the very common practice of checking in unreviewable artifacts into the source tree (e.g., the configure script, malicious binary test artifacts). Reproducible builds guarantee that two people can follow the same instructions and get the same, bit-identical outcome. It does nothing to guarantee that those instructions have not been compromised, and all of the great packaging security failures of my lifetime that I can think of have relied on those instructions being compromised (e.g., xz utils, Debian OpenSSL keygen issues).
    - goodpoint 6mo ago
      No, reproducible builds make such backdoors more difficult to sneak in together with other checks.
    - uecker 6mo ago
      An attack would be far easier without reproducible packages. One could upload a compromised binary to debian by becoming a debian developer, blackmail a debian developer to so, or compromise the computer of a debian developer or the distribution. At the time of xz attack, the package was already reproducible.
- deknos 6mo ago
  "mimimimi". Those people do not care about quality in opensource at all. For longliving software this is very important. Of course, all those javascript and kubernetes packages which are irrelevant in a few years again, might complain, but let them complain.
- ckastner 6mo ago
  > There was no bug or attack on Debian since 2007 that reproducible packages would prevent. I'm reading this as a suggestion that the reproducible builds effort was an ineffective deterrent. However, note that your observation could also be explained by the opposite: the reproducible builds effort was an effective deterrent, so nobody bothered with attempts. > And it just ups the the contribution barrier to Debian higher Until yesterday, the package just got flagged in the tracker, and you could either ignore it, or fix it yourself, or the kind people behind the reproducible builds effort supplied a patch themselves. Now, you can no longer ignore it. But fixes are often trivial. Use a (stable) timestamp provided by the build, seed RNGs with some constant (instead of eg: time), etc. These are best practices anyway.
  - Atotalnoob 6mo ago
    That’s a big logical fallacy, I’m not sure if that’s what you want to go with
  - PunchyHamster 6mo ago
    > However, note that your observation could also be explained by the opposite: the reproducible builds effort was an effective deterrent, so nobody bothered with attempts. There was no attack that reproducible builds would help protect from before 2007 either. > Until yesterday, the package just got flagged in the tracker, and you could either ignore it, or fix it yourself, or the kind people behind the reproducible builds effort supplied a patch themselves. > Now, you can no longer ignore it. But fixes are often trivial. Use a (stable) timestamp provided by the build, seed RNGs with some constant (instead of eg: time), etc. that's the entirety of the problem. App developers don't want to be package experts or build experts. > These are best practices anyway. They are not. They are best practices if you want reproducible builds. They are entirely useless waste of time if you don't care.
    - yjftsjthsd-h 6mo ago
      > that's the entirety of the problem. App developers don't want to be package experts or build experts. App developers and Debian package maintainers are already separate groups.

by uecker 6mo ago

PunchyHamster 6mo ago

There was no bug or attack on Debian since 2007 that reproducible packages would prevent. "Well worth it" is not correct. And it just ups the the contribution barrier to Debian higher, I already heard a lot of people complaining that contributing to Debian is hard and while in past I defended it by "they need all the checks and bounds to make sure packages play with eachother nicely", this is just step that makes it hard for no reason and little benefit.

aborsy 6mo ago
There was perhaps no detected bug or attack. There have most likely been bugs or attacks that reproducible builds would have prevented.
- PunchyHamster 6mo ago
  And you base it on what exactly ? It's "just" making sure the build process is always ordered. If anything it will make attacker's job easier, as Ubuntu package will have same files structured exactly same way as Debian one.
  - yjftsjthsd-h 6mo ago
    > as Ubuntu package will have same files structured exactly same way as Debian one. As opposed to what? If Ubuntu uses the same source, of course they get the same binaries. And if Ubuntu applies patches, they'll get something different. And that's still true.
- CyberDildonics 6mo ago
  There have most likely been bugs or attacks that reproducible builds would have prevented. Like what exactly?
savolai 6mo ago
” If you are wondering why we are doing this at all, then hopefully the Reproducible Builds website will explain why this is useful.” https://reproducible-builds.org/ https://reproducible-builds.org/ Could you perhaps respond to the argumentation here?
- dvogel 6mo ago
  (Not OP, but...) I still fail to see the current value in confirming that a reproducing builder also included the same compromised dependency that I did when I built it. I understand that reproducible builds are guarding against dynamic attacks within build infrastructure. However I just don't see those happening. Compromised source dependencies are a 100x more common problem.
  - deleted 6mo ago
    [deleted]
  - fsflover 6mo ago
    https://en.wikipedia.org/wiki/XZ_Utils_backdoor https://en.wikipedia.org/wiki/XZ_Utils_backdoor
    - mmooss 6mo ago
      https://news.ycombinator.com/item?id=48083768 https://news.ycombinator.com/item?id=48083768
    - PunchyHamster 6mo ago
      that's example of attack reproducible packages do not protect from, why you are linking it ?
      - fsflover 6mo ago
        A distro automatically verifying that installed packages are reproducible would protect the user?
azkalam 6mo ago
Reproducible builds reduce the need for trusted parties. Have many organizations produce the binaries independently and post the arifacts. Once n of m parties agree on the arifact hash, take that as the trusted build. If every party reaches a different hash then we cannot build consensus.
- sgc 6mo ago
  To move away from organizational dependence, there should be an installable project for debian where I can dedicate some configurable small percentage of my compute when idle to reproducibly building debian components to make a robust verification system, starting with the most critical code. Obviously, it would be a ton of work to make such a system resistant to gaming by malicious actors (see GNU Guix for useful efforts), but it would provide valuable diversity in architecture and (political or other) control. It would be even cooler if we could have independent projects that could run on various distros and OS, and build packages for any of them. Having packages for bsd verified on linux and vice-versa with statistical logging (this code has been verified x times on y OSes) would be reassuring.
  - PunchyHamster 6mo ago
    I think that project is called Ubuntu.
    - saghm 6mo ago
      Building Ubuntu does not produce identical binaries to Debian, so no, that's not what they're asking for
    - sgc 6mo ago
      I don't know of anything Ubuntu is doing that is significantly beyond what Debian is doing in this regard, nor that they have a distributed reproduction system set up???
eptcyka 6mo ago
It makes shipping backdoors a whole lot harder, yes.
- zaphirplane 6mo ago
  Hmm, it prevents Trojan binaries which is a small subset of backdoor IMHO. Defense in depth obviously is a good thing
- PunchyHamster 6mo ago
  Unless someone spins entirely separate infrastructure dedicated just for verifying Debian packages, it doesn't.
MomsAVoxell 6mo ago
Reproducible builds are applicable not only to respond to ‘attacks’, a subject you seem to be bikeshedding, but also for other reasons too. Anyone having to maintain a code base or a distributed fleet of devices will gain from this decision, immensely, as their operational periods come and go. Reproducible builds are about longevity as much as they are about security. Please don’t make bold claims about ‘no reason and little benefit’ while demonstrating ignorance of this hard fact: reproducible builds should have been the norm, in computing, from the get-go.
- bluGill 6mo ago
  I longevity is harmed though. Your certs need to expire in a few years we think that your toolchain will not be downloadable.
  - MomsAVoxell 6mo ago
    Those problems need to be solved as well.
    - bluGill 6mo ago
      I don't think they do, actually. Longevity sounds good, but in reality anything that's old probably has critical security holes and so you shouldn't use it anyway.
      - iamnothere 6mo ago
        A warning is sufficient. Old tech should continue to work, for preservation and archival reasons.
        PunchyHamster 6mo ago
        just archiving binary artifacts and source packages is enough Reproducibility adds nothing here
benregenspan 6mo ago
Is the "Jia Tan" XZ Utils compromise not a good example? That relied on code snuck into a release that was not in source. (It was caught before being promoted into a stable Debian release, yes, but this sort of relied on a happy accident, too close for comfort)
- chuckadams 6mo ago
  The xz hack was still reproducible, because it was included in the distribution archive which did not match the upstream source -- even then, it was so obfuscated it likely would have gone unnoticed, but nevertheless it only lived in the uploaded tarball and not in the repo. Reproducibility is a good thing, but the next step is build provenance. Still, lots of good non-security benefits to reproducible builds too.
- jcranmer 6mo ago
  The xz utils compromise is a very good example... of why reproducible packages doesn't actually solve anything security-wise! The backdoor relied first on a difference between building a package in a packaging environment versus building the package on your own. And also, it relied on the very common practice of checking in unreviewable artifacts into the source tree (e.g., the configure script, malicious binary test artifacts). Reproducible builds guarantee that two people can follow the same instructions and get the same, bit-identical outcome. It does nothing to guarantee that those instructions have not been compromised, and all of the great packaging security failures of my lifetime that I can think of have relied on those instructions being compromised (e.g., xz utils, Debian OpenSSL keygen issues).
  - goodpoint 6mo ago
    No, reproducible builds make such backdoors more difficult to sneak in together with other checks.
  - uecker 6mo ago
    An attack would be far easier without reproducible packages. One could upload a compromised binary to debian by becoming a debian developer, blackmail a debian developer to so, or compromise the computer of a debian developer or the distribution. At the time of xz attack, the package was already reproducible.
deknos 6mo ago
"mimimimi". Those people do not care about quality in opensource at all. For longliving software this is very important. Of course, all those javascript and kubernetes packages which are irrelevant in a few years again, might complain, but let them complain.
ckastner 6mo ago
> There was no bug or attack on Debian since 2007 that reproducible packages would prevent. I'm reading this as a suggestion that the reproducible builds effort was an ineffective deterrent. However, note that your observation could also be explained by the opposite: the reproducible builds effort was an effective deterrent, so nobody bothered with attempts. > And it just ups the the contribution barrier to Debian higher Until yesterday, the package just got flagged in the tracker, and you could either ignore it, or fix it yourself, or the kind people behind the reproducible builds effort supplied a patch themselves. Now, you can no longer ignore it. But fixes are often trivial. Use a (stable) timestamp provided by the build, seed RNGs with some constant (instead of eg: time), etc. These are best practices anyway.
- Atotalnoob 6mo ago
  That’s a big logical fallacy, I’m not sure if that’s what you want to go with
- PunchyHamster 6mo ago
  > However, note that your observation could also be explained by the opposite: the reproducible builds effort was an effective deterrent, so nobody bothered with attempts. There was no attack that reproducible builds would help protect from before 2007 either. > Until yesterday, the package just got flagged in the tracker, and you could either ignore it, or fix it yourself, or the kind people behind the reproducible builds effort supplied a patch themselves. > Now, you can no longer ignore it. But fixes are often trivial. Use a (stable) timestamp provided by the build, seed RNGs with some constant (instead of eg: time), etc. that's the entirety of the problem. App developers don't want to be package experts or build experts. > These are best practices anyway. They are not. They are best practices if you want reproducible builds. They are entirely useless waste of time if you don't care.
  - yjftsjthsd-h 6mo ago
    > that's the entirety of the problem. App developers don't want to be package experts or build experts. App developers and Debian package maintainers are already separate groups.