28 ms·

Debian must ship reproducible packages

372 points by robalni 5mo ago

blueflow 5mo ago
zero improvement on end-user experience. does not solve supply chain issues, debian package will reproducabily contain the malware from upstream.
- rlpb 5mo ago
  Debian has had a better "software supply chain" posture than any other player in the ecosystem since before the turn of the century. While we all face the risk of malware from upstream, Debian is the least at risk of being affected by it. See for example the stream of issues from npm et al. None of it has affected Debian.
  - alkindiffie 5mo ago
    > for example the stream of issues from npm et al. Curious, what distros where affected by npm supply chain attacks?
    - throw_a_grenade 5mo ago
      It's npm that's affected, therefore it's not even considered when choosing language/ecosystem for writing distro tools. You'll find no sane distro writing package manager in javascript precisely to avoid this joke of a supply chain.
      - skydhash 5mo ago
        I quite like the OpenBSD approach to Go and Rust projects in ports. They store all the dependencies and their hashes in the build recipe, not trusting the project ones. And they’re more readable. Here is jujutsu’s list of dependencies[0] and their hashes[1]. As an aside, that’s why I don’t like those packages managers. Something like Python’s numpy or lib curl, get sliced into atomic portions. [0]: https://github.com/openbsd/ports/blob/master/devel/jujutsu/crates.inc https://github.com/openbsd/ports/blob/master/devel/jujutsu/c... [1]: https://github.com/openbsd/ports/blob/master/devel/jujutsu/distinfo https://github.com/openbsd/ports/blob/master/devel/jujutsu/d...
      - cxr 5mo ago
        ECMA-262 doesn't require the use of NPM or NodeJS. (In fact, they are at odds, even 10+ years after modules were standardized in ES6.)
  - suprjami 5mo ago
    You do remember the xz-utils backdoor was found in Sid right? https://en.wikipedia.org/wiki/XZ_Utils_backdoor https://en.wikipedia.org/wiki/XZ_Utils_backdoor
    - Dwedit 5mo ago
      It would have been found in a whole lot more places if it hadn't been for that meddling Microsoft employee.
- iveqy 5mo ago
  It does not solve all supply chain issues, it do solve some supply chain issues. Not being able to see if the source code shipped is the same as been used for creating the binary is scary
  - murderfs 5mo ago
    Has there been a single publicly known attack that would have been prevented by this?
    - PunchyHamster 5mo ago
      Zero in Debian. They have enough other procedures to catch it. Less diligent projects had it but there are easier ways to fix it
    - LtWorf 5mo ago
      Several actually. Pypi is regularly targeted in this way.
      - PunchyHamster 5mo ago
        Hasn't happened in Debian
        MomsAVoxell 5mo ago
        “Hasn’t happened” is quite naive. It happens internally - putting unscrupulous code in a company’s distro before torching the place is a surprisingly regular occurrence in places which have long since adopted Debian as a platform host. IT departments around the globe will benefit from this immensely.
        PunchyHamster 5mo ago
        And reproducible builds do not prevent that. The one single fail point they prevent is infected build hosts. That might be some reasonable benefit for the company if it is building it on public architecture, but for projects like Debian that insist build hosts are basically offline (package in, package out with no internet access during build process) it is very fringe benefit.
- quantummagic 5mo ago
  > zero improvement on end-user experience. Maybe not by itself, but it does allow for the ecosystem to be audited, in a way that ultimately benefits the end-user. It really is an important part of a healthy supply chain.
  - miohtama 5mo ago
    [flagged]
    - testdelacc1 5mo ago
      While taking no stance on your statement, I think “fewer” works better in this context than “less”.
      - gjvc 5mo ago
        "fewer" when the measure is discrete, "less" when it is continuous
    - LtWorf 5mo ago
      NK isn't the hostile superpower I'm most concerned about.
  - PunchyHamster 5mo ago
    no problem in Debian since the start of the effort would be solved by reproductible builds This is nice pat yourself on the back achievement for people that prefer security theatre and checking boxes than doing something actually useful, and they wasted thousands man hours of poor victims that had to implement it
- mschuster91 5mo ago
  That's not what reproducible builds aim to prevent, and no one claims that. When upstream pushes bad code, that's on upstream. The thing reproducible builds aim to prevent is Debian or individual developers and system administrators with access rights to binary uploads and signing keys to get forced to sign and upload binary packages by attackers - be these governments (with or without court orders) or criminal organizations. As of now, say if I were an administrator of Debian's CI infrastructure, technically there would be nothing preventing me from running an "extra" job on the CI infrastructure building a package for openssh with a knock-knock backdoor, properly signing it and uploading it to the repository. For someone to spot the attack and differentiate it, they'd have to notice that there is a package in the repository that has no corresponding build logs or has issues otherwise. But with reproducible builds, anyone can set up infrastructure to rebuild Debian packages from source automatically and if there is a mismatch with what is on Debian's repository, raise alarm bells.
  - ownagefool 5mo ago
    Reproducible builds shows that, within a specific configuration, the code produced the binary, regardless of who signed or published it. Indeed, this could mitigate an attacker replacing the binary with something that's not produced from the code, but it does not mitigate the tool chain or code itself containing the exploit, creating a malicious binary.
- shevy-java 5mo ago
  Well - reproducible also means code guarantee. It may not improve an end-user experience directly, but you get an extra quality control step, as guarantee, here. I think reproducibility is great. If we can achieve that, it should be achieved. See also NixOS; it can guarantee that snapshot xyz works, not just for one user, but ALL users. I see it as hopping from guarantee to guarantee. That's actually a good thing in the long run. Just think differently here.
- hiAndrewQuinn 5mo ago
  This is some of the best news I've heard recently when it comes to figuring out how to produce high quality Software Bills of Materials for the upcoming EU Cyber Resilience Act, for what it's worth. Reproducible packages are actually worth a great deal when you are selling products with digital elements. Much easier to scan through, audit, etc. with confidence.
- atoav 5mo ago
  If you find yourself holding opinions of the kind: "If it can't be made perfect, it shouldn't be changed at all?" you may want to consider that most things that work well today were incrementally improved. Reproducable builds are not solving all issues as you rightly observed, but they can be a stepping stone (or even a pre-condition) for further measures.
- otabdeveloper4 5mo ago
  > zero improvement on end-user experience The end-user experience is that now you can host your Debian binaries in caches and CDNs without worrying about supply chain hackers. You can verify that file hashes match the ones on Debian's website and sleep much better at night. If you don't trust Debian's website then you can rebuild yourself and check if Debian has been compromised.
  - tremon 5mo ago
    You could already do that since Debian cryptographically signs all its package indexes, and the indexes contain the hash of all packages. The additional guarantee that reproducible builds bring is that you can re-build the packages in your own controlled environment and verify that the resulting package is bit-for-bit identical to what Debian offers.
    - otabdeveloper4 5mo ago
      Cryptographic signatures only protect against MitM (something extremely rare in the real world) and do nothing against compromised Debian infrastructure and supply chains (the real attack vector 99% of the time). Reproducible builds protect against all attacks.
      - tremon 5mo ago
        > Reproducible builds protect against all attacks. Generic statements like this are always false. As a simple rebuttal, reproducible builds do not protect against source-level attacks such as intentional backdoors or disabled/obfuscated access checks. In fact, I'd say that reproducible builds protect against one class of attacks only: compromise of the build infrastructure.
- MomsAVoxell 5mo ago
  Who is this mythical end user? Reproducible builds are good for everyone - not just the average joe.
shevy-java 5mo ago
A small step for debian, giant leap for mankind.
- stingraycharles 5mo ago
  As someone who recently spent a lot of time on making a large C++ program entirely reproducible on 4 different OS’es, one cannot understate just how many tiny details matter here.
  - gjvc 5mo ago
    "overstate"
    - stingraycharles 5mo ago
      Whoops, yes. Well I hope the point came across anyway.
      - cyh555 5mo ago
        it's funny that as a non-native speaker, I have to check with Gemini about how "cannot overstate" is used I also asked Gemini whether we express ourselves that way in my mother tongue (Mandarin), and yes, we do, but it came off as being too formal way of speaking. We don't normally use it (I'm not from China/Taiwan though)
jaypatelani 5mo ago
Good thing. NetBSD has fully reproductible build since 2017. https://blog.netbsd.org/tnf/entry/netbsd_fully_reproducible_builds https://blog.netbsd.org/tnf/entry/netbsd_fully_reproducible_...
- idoubtit 5mo ago
  As pointed in your link, NetBSD achieved this with some help from Debian. If I understand correctly, it's not that NetBSD tried harder, it's that their problem was easier: fewer packages which change less (they still use CVS, "stability" is an understatement!). BTW, most Debian packages have reproducible builds. Those which have not (I'd say 5%) are shown in orange in the graph there: https://wiki.debian.org/ReproducibleBuilds https://wiki.debian.org/ReproducibleBuilds
  - kakwa_ 5mo ago
    Also, the *BSD are structured somewhat differently to a Linux distro. It's not like the Linux world where you have distinct projects like the Kernel, GNU, OpenSSL, and then it's the distributions job to assemble everything. In the BSD projects, the scope is developing and distributing an entire base system, i.e., the kernel but also the libc, the shell/all posix utilities, and a few third parties like OpenSSH (which are usually "softforked"). It's quite visible in the sources, it's a lot more than just a kernel: https://github.com/NetBSD/src https://github.com/NetBSD/src Additional packages you could get from pkg_in/pkgsrc (NetBSD), pkg-ng/ports (FreeBSD) or pkg_add (OpenBSD) are clearly distinct from the base system, installed in a dedicated subtree (/usr/src in NetBSD, /usr/local/ OpenBSD/FreeBSD), and provided in a best effort manner. The reproducible build target was almost certainly only for the base system, which is a few percent of what Debian tries to achieve, and on which NetBSD has a tighter control over (developer + distributor instead of downstream assembler+distributor). A reproducible base system is useful, but given how quickly you typically need to install packages from pkgsrc, it's not quite enough.
  - mmooss 5mo ago
    > it's not that NetBSD tried harder, it's that their problem was easier: fewer packages which change less Maybe that's trying harder on design rather than trying to remedy the consequences later.
kkfx 5mo ago
Debian, like any other legacy distro, mush became declarative, because the '80s model of manual deploy and the absurd pain of D/I and Preseed must end.
- kakwa_ 5mo ago
  In the end, Nix is just a thin veneer on this stuff. Given how many quick & dirty sed patching or exec commands I've seen in the few nix package/modules I've read, I would not exactly bet my life on it being completely idempotent & reproducible.
  - kkfx 5mo ago
    it's the best option after IllumOS (OpenSolaris) IPS integrated with ZFS. Far less powerful not imposing zfs (only well supported for root, swap, encryption etc), so not integrated in the package system and bootloader management (BEs, Boot Environments). It's not reproducible bit by bit, it fetch the current version of anything, but it's still easy to reproduce enough, stable enough and complete enough, while classic distros need a fresh install every major release or facing issues an keeping a system in unknown state for long until it explode.
- suprjami 5mo ago
  bootcrew have bootc Containerfiles for Debian, Ubuntu, Arch, and openSUSE: https://github.com/bootcrew/mono https://github.com/bootcrew/mono
- farfatched 5mo ago
  I've been 100% on NixOS on many years, but it's Debian that really drove this project. They're still a pragmatic choice for many usecases.
Zopieux 5mo ago
A great milestone, congrats Debian on taking a stance and holding high standards for yourself, especially in the current era.
inglor_cz 5mo ago
Has anyone fought Microsoft Visual Studio successfully to produce reproducible builds of C++ programs? From what I have heard, it is one of the worst contexts to do it.
- azkalam 5mo ago
  Probably easiest way is to use Bazel to leverage the effort that has gone in there
- einpoklum 5mo ago
  Well, you can't build MSVS yourself, reproducibly or otherwise, so this is a less appealing endeavor I would think.
- Dwedit 5mo ago
  It's that RICH header that you need to exclude. I just tested my copy of MSVC 2019, and `/emittoolversioninfo:no` will exclude the RICH header from the binary. Supposedly also works in MSVC 2022. The build timestamps in the PE header and export table are also a problem as well.
pixel_popping 5mo ago
Forbidden You don't have permission to access this resource. Apache Server at lists.debian.org Port 443 :/
- ameliaquining 5mo ago
  I can see it just fine; maybe an overzealous firewall thinks you're a bot? At any rate, the Wayback Machine has it: https://web.archive.org/web/20260510074120/https://lists.debian.org/debian-devel-announce/2026/05/msg00001.html https://web.archive.org/web/20260510074120/https://lists.deb...
  - baranul 5mo ago
    Unfortunately, many of these "protections" don't know what is a bot or a human. Many clueless websites are often just blocking huge swaths of legitimate readers and customers.
  - pixel_popping 5mo ago
    Why would you block access to a static page, even Bots, what's the point? I'm not a bot, very typical non-privacy setup (Firefox, Linux, VPN) for personal usage. It does work with my privacy/scrapping setup (residential proxy, spoofed fingerprints, Qubes and so on), great job debian.
perlgeek 5mo ago
https://wiki.debian.org/ReproducibleBuilds https://wiki.debian.org/ReproducibleBuilds has some more infos; some is outdated, but it also has a chart showing how many packages are built in the CI, and how many of those are reproducible builds. (Orange = FTBR = "failed to build reproducibly") I'm not good at reading numbers from charts, but I'd guess it's a few percent (4-5ish?).
- bpavuk 5mo ago
  all I get is this: > Forbidden > <p>You are not allowed to access this!</p> (yes, with HTML tags on display) :) EDIT: I also found a "I Challenge Thee" page in history. did I just get blocked by antibot measures? why???
  - unleaded 5mo ago
    Do you have JavaScript disabled? They put one of those anti-scraper things on it.
    - bpavuk 5mo ago
      nope, it's enabled. I can pass Cloudflare, reCaptcha, whatever Microsoft is doing, and Annubis, but Debian caught me off-guard
idovmamane 5mo ago
[dead]
uecker 5mo ago
This is a huge achievement for Debian and the free software world. It took a while though until this was understood. In 2007 when pointing out on debian-devel that this is needed, I was still told what huge waste of time this would be. And indeed it took a huge amount of work by many people to get there, but it is well worth it.
- PunchyHamster 5mo ago
  There was no bug or attack on Debian since 2007 that reproducible packages would prevent. "Well worth it" is not correct. And it just ups the the contribution barrier to Debian higher, I already heard a lot of people complaining that contributing to Debian is hard and while in past I defended it by "they need all the checks and bounds to make sure packages play with eachother nicely", this is just step that makes it hard for no reason and little benefit.
  - aborsy 5mo ago
    There was perhaps no detected bug or attack. There have most likely been bugs or attacks that reproducible builds would have prevented.
    - PunchyHamster 5mo ago
      And you base it on what exactly ? It's "just" making sure the build process is always ordered. If anything it will make attacker's job easier, as Ubuntu package will have same files structured exactly same way as Debian one.
      - yjftsjthsd-h 5mo ago
        > as Ubuntu package will have same files structured exactly same way as Debian one. As opposed to what? If Ubuntu uses the same source, of course they get the same binaries. And if Ubuntu applies patches, they'll get something different. And that's still true.
    - CyberDildonics 5mo ago
charcircuit 5mo ago
So much time has been wasted on reproducible builds which could have better spent on securing more important parts of Debian. Practically minor changes like a build timestamp being different is not an issue.
- Hendrikto 5mo ago
  It allows verifying that the binaries actually match the source, which is extremely valuable.
  - charcircuit 5mo ago
    Bit for bit matching is not required for that.
    - Hendrikto 5mo ago
      It makes it much simpler and more robust though. Also, it allows for content addressing a la Nix, among other benefits.
      - charcircuit 5mo ago
        I reject that those benefits are actually that useful compared to the effort needed to do so. You can content addressing without reproducible builds. You just have a canonical version, which typically is built by the application developer.
- farfatched 5mo ago
  Yes, making sure build timestamps are reproducible isn't a security win. What is a win is that two independent parties can run the same build, and get the same binaries. This is important because it removes trust from builders: anyone can verify their output. It just so happens that unimportant things like build versions impede that.
  - charcircuit 5mo ago
micw 5mo ago
I wonder why this is a thing nowadays. I use yocto for embedded devices and it was almost a no-brainer to implement reproducible builds. I can also easily enable Debian package management, so everything is already available.
- MomsAVoxell 5mo ago
  What do you mean why is it a thing nowadays? Reproducible builds are an essential method in industrial computing - Debian isn’t at the forefront of this, it is merely adopting industry wide techniques also applied to other operating systems in use in long-term and safety-related applications. Certainly, a lot of the hard work of the Yocto and Debian developers is already in your hands. What is interesting is that this is now being applied in a more forward-focused policy by the Debian developers, that it will now be the norm rather than an option…
- dezgeg 5mo ago
  Did you actively verify that your builds were bit-reproducible?
deleted 5mo ago
[deleted]
einpoklum 5mo ago
Debian must ship packages without the hard dependence on systemd.
suprjami 5mo ago
I am always surprised Debian are leading this and not the commercial vendors. You'd think big organisations paying for RHEL and Ubuntu would be beating down the door for verifiable binaries.
- tremon 5mo ago
  If a competitor can prove that their packages are bit-for-bit identical to what a big organization is shipping, that allows the competitor to benefit from the security assurances of the big org. This is great for software freedom, not so great for wannabe monopolists.
- jorams 5mo ago
  Reproducible builds exist to reduce the need for trust, while commercial vendors are in the business of selling trust.
Hendrikto 5mo ago
Why the fuck does that site break the back button? DO NOT do that.
- em-bee 5mo ago
  since there is no other way to reach you please allow me to use this off topic message to let you know that there is a response to your comments on the gnupg discussion from two weeks ago.
peterhadlaw 5mo ago
[flagged]
TacticalCoder 5mo ago
What people really don't understand about reproducible builds is that they're not a guarantee that there's no backdoor. They're a guarantee that if there's a backdoor, it's reproducible 100% of the time. This is a godsend for white hats fighting the good fight. And, as a side note, it's strongarming vs the bad guys: "Would be too bad if we could reproduce your shiny exploit 100% of the time wouldn't it!?". Note that we should go further (but it's a bit orthogonal to reproducible builds): builds of the final binary/package should happen by first entirely discarding all files not necessary for the final build (like all test cases and all test assets). The build should literally happen in an environment that gets rid of those (after, of course, having test in another environment that all tests cases succeed): if I'm not mistaken get rid of test assets would have stopped Jia Tan's XZ backdoor attempt dead in its track (for example). Because IIRC there were binary data part of the backdoor hidden in some asset only used by test cases. P.S: as a bonus they also allow to detect bit-flips (I'm not saying there aren't other ways to detect bit-flips: what I'm saying is that if you have deterministic builds anyway and something doesn't reproduce correctly due to a flipped-bit, it's going to be noticed).
- sieabahlpark 5mo ago
  [dead]
deleted 5mo ago
[deleted]
tofflos 5mo ago
amd64 forky reproduced: 97.02% good: 17586 bad: 511 fail: 30 unknown: 0 This, statistics for other architectures, and the reasons for unreproducibility can be found at https://reproduce.debian.net https://reproduce.debian.net.
amelius 5mo ago
That's cool but I'm honestly a bit disappointed in how apt refused to embrace/support both the container and AI/GPU aspects of computing. Are we going to see some changes there?
- yjftsjthsd-h 5mo ago
  Those seem like unrelated things? I can imagine ways for apt to integrate with containers, but what would it possibly do for AI or GPU other than delivering packages like it already does?
- Arrowmaster 5mo ago
  What exactly are you talking about? Those don't seem related.
kkyktkrkekk 5mo ago
”Optimize the code for 5 seconds”, as many compilers, including vc++ on windows did, was probably one of the dumbest thing ever invented. It meant that the binaries became more optimized when building on faster computers.
jgneff 5mo ago
I'm so happy to see this change. I got involved with reproducible builds in 2021 after reading in horror about the SolarWinds attack. [1] I think Magnus Ihse Bursie said it best while working on reproducible builds of OpenJDK: "If you were to ask me, the fact that compilers and build tools ever started to produce non-deterministic output has been a bug from day one." [2] [1] https://www.linux.com/news/preventing-supply-chain-attacks-like-solarwinds/ https://www.linux.com/news/preventing-supply-chain-attacks-l... [2] https://github.com/openjdk/jdk/pull/9152#issue-1270543997 https://github.com/openjdk/jdk/pull/9152#issue-1270543997
rurban 5mo ago
... and most of this work is done by other distros and maintainers. Starting with binutils
casey2 5mo ago
This fights against "opensource-washing" which is the practice of large companies claiming to release open source code, but the compile takes so long (as well as being overly-convoluted) that most people and many distros can't afford to maintain the package. It feels like AI and traditional software are converging in complexity.
rurban 5mo ago
So these are broken on amd64. Debian arm64/forky rebuilderd stats https://reproduce.debian.net/arm64/stats/forky/ https://reproduce.debian.net/arm64/stats/forky/ Most with failed to reproduce: NT_GNU_BUILD_ID. The others on some other bits. Mostly timestamps or hashes I assume

372 points by robalni 5mo ago

zero improvement on end-user experience. does not solve supply chain issues, debian package will reproducabily contain the malware from upstream.

rlpb 5mo ago
Debian has had a better "software supply chain" posture than any other player in the ecosystem since before the turn of the century. While we all face the risk of malware from upstream, Debian is the least at risk of being affected by it. See for example the stream of issues from npm et al. None of it has affected Debian.
- alkindiffie 5mo ago
  > for example the stream of issues from npm et al. Curious, what distros where affected by npm supply chain attacks?
  - throw_a_grenade 5mo ago
    It's npm that's affected, therefore it's not even considered when choosing language/ecosystem for writing distro tools. You'll find no sane distro writing package manager in javascript precisely to avoid this joke of a supply chain.
    - skydhash 5mo ago
      I quite like the OpenBSD approach to Go and Rust projects in ports. They store all the dependencies and their hashes in the build recipe, not trusting the project ones. And they’re more readable. Here is jujutsu’s list of dependencies[0] and their hashes[1]. As an aside, that’s why I don’t like those packages managers. Something like Python’s numpy or lib curl, get sliced into atomic portions. [0]: https://github.com/openbsd/ports/blob/master/devel/jujutsu/crates.inc https://github.com/openbsd/ports/blob/master/devel/jujutsu/c... [1]: https://github.com/openbsd/ports/blob/master/devel/jujutsu/distinfo https://github.com/openbsd/ports/blob/master/devel/jujutsu/d...
    - cxr 5mo ago
      ECMA-262 doesn't require the use of NPM or NodeJS. (In fact, they are at odds, even 10+ years after modules were standardized in ES6.)
- suprjami 5mo ago
  You do remember the xz-utils backdoor was found in Sid right? https://en.wikipedia.org/wiki/XZ_Utils_backdoor https://en.wikipedia.org/wiki/XZ_Utils_backdoor
  - Dwedit 5mo ago
    It would have been found in a whole lot more places if it hadn't been for that meddling Microsoft employee.
iveqy 5mo ago
It does not solve all supply chain issues, it do solve some supply chain issues. Not being able to see if the source code shipped is the same as been used for creating the binary is scary
- murderfs 5mo ago
  Has there been a single publicly known attack that would have been prevented by this?
  - PunchyHamster 5mo ago
    Zero in Debian. They have enough other procedures to catch it. Less diligent projects had it but there are easier ways to fix it
  - LtWorf 5mo ago
    Several actually. Pypi is regularly targeted in this way.
    - PunchyHamster 5mo ago
      Hasn't happened in Debian
      - MomsAVoxell 5mo ago
        “Hasn’t happened” is quite naive. It happens internally - putting unscrupulous code in a company’s distro before torching the place is a surprisingly regular occurrence in places which have long since adopted Debian as a platform host. IT departments around the globe will benefit from this immensely.
        PunchyHamster 5mo ago
        And reproducible builds do not prevent that. The one single fail point they prevent is infected build hosts. That might be some reasonable benefit for the company if it is building it on public architecture, but for projects like Debian that insist build hosts are basically offline (package in, package out with no internet access during build process) it is very fringe benefit.
quantummagic 5mo ago
> zero improvement on end-user experience. Maybe not by itself, but it does allow for the ecosystem to be audited, in a way that ultimately benefits the end-user. It really is an important part of a healthy supply chain.
- miohtama 5mo ago
  [flagged]
  - testdelacc1 5mo ago
    While taking no stance on your statement, I think “fewer” works better in this context than “less”.
    - gjvc 5mo ago
      "fewer" when the measure is discrete, "less" when it is continuous
  - LtWorf 5mo ago
    NK isn't the hostile superpower I'm most concerned about.
- PunchyHamster 5mo ago
  no problem in Debian since the start of the effort would be solved by reproductible builds This is nice pat yourself on the back achievement for people that prefer security theatre and checking boxes than doing something actually useful, and they wasted thousands man hours of poor victims that had to implement it
mschuster91 5mo ago
That's not what reproducible builds aim to prevent, and no one claims that. When upstream pushes bad code, that's on upstream. The thing reproducible builds aim to prevent is Debian or individual developers and system administrators with access rights to binary uploads and signing keys to get forced to sign and upload binary packages by attackers - be these governments (with or without court orders) or criminal organizations. As of now, say if I were an administrator of Debian's CI infrastructure, technically there would be nothing preventing me from running an "extra" job on the CI infrastructure building a package for openssh with a knock-knock backdoor, properly signing it and uploading it to the repository. For someone to spot the attack and differentiate it, they'd have to notice that there is a package in the repository that has no corresponding build logs or has issues otherwise. But with reproducible builds, anyone can set up infrastructure to rebuild Debian packages from source automatically and if there is a mismatch with what is on Debian's repository, raise alarm bells.
- ownagefool 5mo ago
  Reproducible builds shows that, within a specific configuration, the code produced the binary, regardless of who signed or published it. Indeed, this could mitigate an attacker replacing the binary with something that's not produced from the code, but it does not mitigate the tool chain or code itself containing the exploit, creating a malicious binary.
shevy-java 5mo ago
Well - reproducible also means code guarantee. It may not improve an end-user experience directly, but you get an extra quality control step, as guarantee, here. I think reproducibility is great. If we can achieve that, it should be achieved. See also NixOS; it can guarantee that snapshot xyz works, not just for one user, but ALL users. I see it as hopping from guarantee to guarantee. That's actually a good thing in the long run. Just think differently here.
hiAndrewQuinn 5mo ago
This is some of the best news I've heard recently when it comes to figuring out how to produce high quality Software Bills of Materials for the upcoming EU Cyber Resilience Act, for what it's worth. Reproducible packages are actually worth a great deal when you are selling products with digital elements. Much easier to scan through, audit, etc. with confidence.
atoav 5mo ago
If you find yourself holding opinions of the kind: "If it can't be made perfect, it shouldn't be changed at all?" you may want to consider that most things that work well today were incrementally improved. Reproducable builds are not solving all issues as you rightly observed, but they can be a stepping stone (or even a pre-condition) for further measures.
otabdeveloper4 5mo ago
> zero improvement on end-user experience The end-user experience is that now you can host your Debian binaries in caches and CDNs without worrying about supply chain hackers. You can verify that file hashes match the ones on Debian's website and sleep much better at night. If you don't trust Debian's website then you can rebuild yourself and check if Debian has been compromised.
- tremon 5mo ago
  You could already do that since Debian cryptographically signs all its package indexes, and the indexes contain the hash of all packages. The additional guarantee that reproducible builds bring is that you can re-build the packages in your own controlled environment and verify that the resulting package is bit-for-bit identical to what Debian offers.
  - otabdeveloper4 5mo ago
    Cryptographic signatures only protect against MitM (something extremely rare in the real world) and do nothing against compromised Debian infrastructure and supply chains (the real attack vector 99% of the time). Reproducible builds protect against all attacks.
    - tremon 5mo ago
      > Reproducible builds protect against all attacks. Generic statements like this are always false. As a simple rebuttal, reproducible builds do not protect against source-level attacks such as intentional backdoors or disabled/obfuscated access checks. In fact, I'd say that reproducible builds protect against one class of attacks only: compromise of the build infrastructure.
MomsAVoxell 5mo ago
Who is this mythical end user? Reproducible builds are good for everyone - not just the average joe.

shevy-java 5mo ago

A small step for debian, giant leap for mankind.

stingraycharles 5mo ago
As someone who recently spent a lot of time on making a large C++ program entirely reproducible on 4 different OS’es, one cannot understate just how many tiny details matter here.
- gjvc 5mo ago
  "overstate"
  - stingraycharles 5mo ago
    Whoops, yes. Well I hope the point came across anyway.
    - cyh555 5mo ago
      it's funny that as a non-native speaker, I have to check with Gemini about how "cannot overstate" is used I also asked Gemini whether we express ourselves that way in my mother tongue (Mandarin), and yes, we do, but it came off as being too formal way of speaking. We don't normally use it (I'm not from China/Taiwan though)

jaypatelani 5mo ago

Good thing. NetBSD has fully reproductible build since 2017. https://blog.netbsd.org/tnf/entry/netbsd_fully_reproducible_builds https://blog.netbsd.org/tnf/entry/netbsd_fully_reproducible_...

idoubtit 5mo ago
As pointed in your link, NetBSD achieved this with some help from Debian. If I understand correctly, it's not that NetBSD tried harder, it's that their problem was easier: fewer packages which change less (they still use CVS, "stability" is an understatement!). BTW, most Debian packages have reproducible builds. Those which have not (I'd say 5%) are shown in orange in the graph there: https://wiki.debian.org/ReproducibleBuilds https://wiki.debian.org/ReproducibleBuilds
- kakwa_ 5mo ago
  Also, the *BSD are structured somewhat differently to a Linux distro. It's not like the Linux world where you have distinct projects like the Kernel, GNU, OpenSSL, and then it's the distributions job to assemble everything. In the BSD projects, the scope is developing and distributing an entire base system, i.e., the kernel but also the libc, the shell/all posix utilities, and a few third parties like OpenSSH (which are usually "softforked"). It's quite visible in the sources, it's a lot more than just a kernel: https://github.com/NetBSD/src https://github.com/NetBSD/src Additional packages you could get from pkg_in/pkgsrc (NetBSD), pkg-ng/ports (FreeBSD) or pkg_add (OpenBSD) are clearly distinct from the base system, installed in a dedicated subtree (/usr/src in NetBSD, /usr/local/ OpenBSD/FreeBSD), and provided in a best effort manner. The reproducible build target was almost certainly only for the base system, which is a few percent of what Debian tries to achieve, and on which NetBSD has a tighter control over (developer + distributor instead of downstream assembler+distributor). A reproducible base system is useful, but given how quickly you typically need to install packages from pkgsrc, it's not quite enough.
- mmooss 5mo ago
  > it's not that NetBSD tried harder, it's that their problem was easier: fewer packages which change less Maybe that's trying harder on design rather than trying to remedy the consequences later.

kkfx 5mo ago

Debian, like any other legacy distro, mush became declarative, because the '80s model of manual deploy and the absurd pain of D/I and Preseed must end.

kakwa_ 5mo ago
In the end, Nix is just a thin veneer on this stuff. Given how many quick & dirty sed patching or exec commands I've seen in the few nix package/modules I've read, I would not exactly bet my life on it being completely idempotent & reproducible.
- kkfx 5mo ago
  it's the best option after IllumOS (OpenSolaris) IPS integrated with ZFS. Far less powerful not imposing zfs (only well supported for root, swap, encryption etc), so not integrated in the package system and bootloader management (BEs, Boot Environments). It's not reproducible bit by bit, it fetch the current version of anything, but it's still easy to reproduce enough, stable enough and complete enough, while classic distros need a fresh install every major release or facing issues an keeping a system in unknown state for long until it explode.
suprjami 5mo ago
bootcrew have bootc Containerfiles for Debian, Ubuntu, Arch, and openSUSE: https://github.com/bootcrew/mono https://github.com/bootcrew/mono
farfatched 5mo ago
I've been 100% on NixOS on many years, but it's Debian that really drove this project. They're still a pragmatic choice for many usecases.

Zopieux 5mo ago

A great milestone, congrats Debian on taking a stance and holding high standards for yourself, especially in the current era.

inglor_cz 5mo ago

Has anyone fought Microsoft Visual Studio successfully to produce reproducible builds of C++ programs? From what I have heard, it is one of the worst contexts to do it.

azkalam 5mo ago
Probably easiest way is to use Bazel to leverage the effort that has gone in there
einpoklum 5mo ago
Well, you can't build MSVS yourself, reproducibly or otherwise, so this is a less appealing endeavor I would think.
Dwedit 5mo ago
It's that RICH header that you need to exclude. I just tested my copy of MSVC 2019, and `/emittoolversioninfo:no` will exclude the RICH header from the binary. Supposedly also works in MSVC 2022. The build timestamps in the PE header and export table are also a problem as well.

pixel_popping 5mo ago

Forbidden You don't have permission to access this resource. Apache Server at lists.debian.org Port 443 :/

ameliaquining 5mo ago
I can see it just fine; maybe an overzealous firewall thinks you're a bot? At any rate, the Wayback Machine has it: https://web.archive.org/web/20260510074120/https://lists.debian.org/debian-devel-announce/2026/05/msg00001.html https://web.archive.org/web/20260510074120/https://lists.deb...
- baranul 5mo ago
  Unfortunately, many of these "protections" don't know what is a bot or a human. Many clueless websites are often just blocking huge swaths of legitimate readers and customers.
- pixel_popping 5mo ago
  Why would you block access to a static page, even Bots, what's the point? I'm not a bot, very typical non-privacy setup (Firefox, Linux, VPN) for personal usage. It does work with my privacy/scrapping setup (residential proxy, spoofed fingerprints, Qubes and so on), great job debian.

perlgeek 5mo ago

https://wiki.debian.org/ReproducibleBuilds https://wiki.debian.org/ReproducibleBuilds has some more infos; some is outdated, but it also has a chart showing how many packages are built in the CI, and how many of those are reproducible builds. (Orange = FTBR = "failed to build reproducibly") I'm not good at reading numbers from charts, but I'd guess it's a few percent (4-5ish?).

bpavuk 5mo ago
all I get is this: > Forbidden > <p>You are not allowed to access this!</p> (yes, with HTML tags on display) :) EDIT: I also found a "I Challenge Thee" page in history. did I just get blocked by antibot measures? why???
- unleaded 5mo ago
  Do you have JavaScript disabled? They put one of those anti-scraper things on it.
  - bpavuk 5mo ago
    nope, it's enabled. I can pass Cloudflare, reCaptcha, whatever Microsoft is doing, and Annubis, but Debian caught me off-guard

idovmamane 5mo ago

[dead]

uecker 5mo ago

This is a huge achievement for Debian and the free software world. It took a while though until this was understood. In 2007 when pointing out on debian-devel that this is needed, I was still told what huge waste of time this would be. And indeed it took a huge amount of work by many people to get there, but it is well worth it.

PunchyHamster 5mo ago
There was no bug or attack on Debian since 2007 that reproducible packages would prevent. "Well worth it" is not correct. And it just ups the the contribution barrier to Debian higher, I already heard a lot of people complaining that contributing to Debian is hard and while in past I defended it by "they need all the checks and bounds to make sure packages play with eachother nicely", this is just step that makes it hard for no reason and little benefit.
- aborsy 5mo ago
  There was perhaps no detected bug or attack. There have most likely been bugs or attacks that reproducible builds would have prevented.
  - PunchyHamster 5mo ago
    And you base it on what exactly ? It's "just" making sure the build process is always ordered. If anything it will make attacker's job easier, as Ubuntu package will have same files structured exactly same way as Debian one.
    - yjftsjthsd-h 5mo ago
      > as Ubuntu package will have same files structured exactly same way as Debian one. As opposed to what? If Ubuntu uses the same source, of course they get the same binaries. And if Ubuntu applies patches, they'll get something different. And that's still true.
  - CyberDildonics 5mo ago

charcircuit 5mo ago

So much time has been wasted on reproducible builds which could have better spent on securing more important parts of Debian. Practically minor changes like a build timestamp being different is not an issue.

Hendrikto 5mo ago
It allows verifying that the binaries actually match the source, which is extremely valuable.
- charcircuit 5mo ago
  Bit for bit matching is not required for that.
  - Hendrikto 5mo ago
    It makes it much simpler and more robust though. Also, it allows for content addressing a la Nix, among other benefits.
    - charcircuit 5mo ago
      I reject that those benefits are actually that useful compared to the effort needed to do so. You can content addressing without reproducible builds. You just have a canonical version, which typically is built by the application developer.
farfatched 5mo ago
Yes, making sure build timestamps are reproducible isn't a security win. What is a win is that two independent parties can run the same build, and get the same binaries. This is important because it removes trust from builders: anyone can verify their output. It just so happens that unimportant things like build versions impede that.
- charcircuit 5mo ago

micw 5mo ago

I wonder why this is a thing nowadays. I use yocto for embedded devices and it was almost a no-brainer to implement reproducible builds. I can also easily enable Debian package management, so everything is already available.

MomsAVoxell 5mo ago
What do you mean why is it a thing nowadays? Reproducible builds are an essential method in industrial computing - Debian isn’t at the forefront of this, it is merely adopting industry wide techniques also applied to other operating systems in use in long-term and safety-related applications. Certainly, a lot of the hard work of the Yocto and Debian developers is already in your hands. What is interesting is that this is now being applied in a more forward-focused policy by the Debian developers, that it will now be the norm rather than an option…
dezgeg 5mo ago
Did you actively verify that your builds were bit-reproducible?

deleted 5mo ago

[deleted]

einpoklum 5mo ago

Debian must ship packages without the hard dependence on systemd.

suprjami 5mo ago

I am always surprised Debian are leading this and not the commercial vendors. You'd think big organisations paying for RHEL and Ubuntu would be beating down the door for verifiable binaries.

tremon 5mo ago
If a competitor can prove that their packages are bit-for-bit identical to what a big organization is shipping, that allows the competitor to benefit from the security assurances of the big org. This is great for software freedom, not so great for wannabe monopolists.
jorams 5mo ago
Reproducible builds exist to reduce the need for trust, while commercial vendors are in the business of selling trust.

Hendrikto 5mo ago

Why the fuck does that site break the back button? DO NOT do that.

em-bee 5mo ago
since there is no other way to reach you please allow me to use this off topic message to let you know that there is a response to your comments on the gnupg discussion from two weeks ago.

peterhadlaw 5mo ago

[flagged]

TacticalCoder 5mo ago

What people really don't understand about reproducible builds is that they're not a guarantee that there's no backdoor. They're a guarantee that if there's a backdoor, it's reproducible 100% of the time. This is a godsend for white hats fighting the good fight. And, as a side note, it's strongarming vs the bad guys: "Would be too bad if we could reproduce your shiny exploit 100% of the time wouldn't it!?". Note that we should go further (but it's a bit orthogonal to reproducible builds): builds of the final binary/package should happen by first entirely discarding all files not necessary for the final build (like all test cases and all test assets). The build should literally happen in an environment that gets rid of those (after, of course, having test in another environment that all tests cases succeed): if I'm not mistaken get rid of test assets would have stopped Jia Tan's XZ backdoor attempt dead in its track (for example). Because IIRC there were binary data part of the backdoor hidden in some asset only used by test cases. P.S: as a bonus they also allow to detect bit-flips (I'm not saying there aren't other ways to detect bit-flips: what I'm saying is that if you have deterministic builds anyway and something doesn't reproduce correctly due to a flipped-bit, it's going to be noticed).

sieabahlpark 5mo ago
[dead]

deleted 5mo ago

[deleted]

tofflos 5mo ago

amd64 forky reproduced: 97.02% good: 17586 bad: 511 fail: 30 unknown: 0 This, statistics for other architectures, and the reasons for unreproducibility can be found at https://reproduce.debian.net https://reproduce.debian.net.

amelius 5mo ago

That's cool but I'm honestly a bit disappointed in how apt refused to embrace/support both the container and AI/GPU aspects of computing. Are we going to see some changes there?

yjftsjthsd-h 5mo ago
Those seem like unrelated things? I can imagine ways for apt to integrate with containers, but what would it possibly do for AI or GPU other than delivering packages like it already does?
Arrowmaster 5mo ago
What exactly are you talking about? Those don't seem related.

kkyktkrkekk 5mo ago

”Optimize the code for 5 seconds”, as many compilers, including vc++ on windows did, was probably one of the dumbest thing ever invented. It meant that the binaries became more optimized when building on faster computers.

jgneff 5mo ago

I'm so happy to see this change. I got involved with reproducible builds in 2021 after reading in horror about the SolarWinds attack. [1] I think Magnus Ihse Bursie said it best while working on reproducible builds of OpenJDK: "If you were to ask me, the fact that compilers and build tools ever started to produce non-deterministic output has been a bug from day one." [2] [1] https://www.linux.com/news/preventing-supply-chain-attacks-like-solarwinds/ https://www.linux.com/news/preventing-supply-chain-attacks-l... [2] https://github.com/openjdk/jdk/pull/9152#issue-1270543997 https://github.com/openjdk/jdk/pull/9152#issue-1270543997

rurban 5mo ago

... and most of this work is done by other distros and maintainers. Starting with binutils

casey2 5mo ago

This fights against "opensource-washing" which is the practice of large companies claiming to release open source code, but the compile takes so long (as well as being overly-convoluted) that most people and many distros can't afford to maintain the package. It feels like AI and traditional software are converging in complexity.

rurban 5mo ago

So these are broken on amd64. Debian arm64/forky rebuilderd stats https://reproduce.debian.net/arm64/stats/forky/ https://reproduce.debian.net/arm64/stats/forky/ Most with failed to reproduce: NT_GNU_BUILD_ID. The others on some other bits. Mostly timestamps or hashes I assume