15 ms·

Debian bookworm live images now reproducible

766 points by bertman 2y ago

imcritic 2y ago
I don't get how someone achieves reproducibility of builds: what about files metadata like creation/modification timestamps? Do they forge them? Or are these data treated as not important enough (like it 2 files with different metadata but identical contents should have the same checksum when hashed)?
- HideousKojima 2y ago
  Those aren't needed to generate a hash of a file. And that metadata isn't part of the file itself (or at least doesn't need to be), it's part of the filesystem or OS
  - imcritic 2y ago
    That's an acceptable answer for the simple case when you distribute just a file, but what if your distribution is something more complex, like an archive with some sub-archives? Metadata in the internal files will affect the checksum of the resulting archive.
    - exe34 2y ago
      unless you fix them to a known epoch.
    - londons_explore 2y ago
      Finding and fixing cases like this are part of what the project has done...
- c0l0 2y ago
  Yes.
- o11c 2y ago
  Timestamps are easiest part - you just set everything according to the chosen epoch. The hard things involve things like unstable hash orderings, non-sorted filesystem listing, parallel execution, address-space randomization, ...
  - koolba 2y ago
    ASLR shouldn’t be an issue unless you intend to capture the entire memory state of the application. It’s an intermediate representation in memory, not an output of any given step of a build. Annoying edge cases come up for things like internal object serialization to sort things like JSON keys in config files.
    - sodality2 2y ago
      Let’s say a compiler is doing something in a multi-threaded manner - isn’t it possible that ASLR would affect the ordering of certain events which could change the compiled output? Sure you could just set threads to 1 but there’s probably some more edge cases in there I haven’t thought of.
      - zamadatix 2y ago
        I think you'd need the compiler to guarantee serialization order of such operations regardless if you used ASLR or not. Otherwise you're just hoping thread scheduling, core clocking, thread memory access, and many other things are the same between every system trying to do a reproducible build. Even setting threads to 1 may not solve that problem class if asynchronous functions/syscalls come into play.
    - cperciva 2y ago
      FreeBSD tripped over an issue recently where a C++ program (I think clang?) used a collection of pointers and output values in an order based on the pointers rather than the values they pointed to. ASLR by itself shouldn't cause reproducibility issues, but it can certainly expose bugs.
      - 2y ago
- purkka 2y ago
  Generally, yes: https://reproducible-builds.org/docs/timestamps/ https://reproducible-builds.org/docs/timestamps/ Since the build is reproducible, it should not matter when it was built. If you want to trace a build back to its source, there are much better ways than a timestamp.
  - ryandrake 2y ago
    C compilers offer __DATE__ and __TIME__ macros, which expand to string constants that describe the date and time that the preprocessor was invoked. Any code using these would have different strings each time it was built, and would need to be modified. I can't think of a good reason for them to be used in an actual production program, but for whatever reason, they exist.
    - fmbb 2y ago
      Toolchains for reproducible software likely let you set these values, or ensure they are 1970-01-01 00:00:00
      - mikepurvis 2y ago
        Nix sets everything to the epoch, although I believe Debian's approach is to just use the date of the newest file in the dsc tarballs.
        yjftsjthsd-h 2y ago
        Nix can also set it to things other than 0; I think my favorite is to set it by the time of the commit from which you're building.
        deleted 2y ago
        [deleted]
        terinjokes 2y ago
        Which is also used when the contents of a derivation will be included in a zip file. The Unix epoch is about a decade older than the zip epoch.
- deleted 2y ago
  [deleted]
- deleted 2y ago
  [deleted]
- jzb 2y ago
  Debian uses a tool called `strip-nondeterminism` to help with this in part: https://salsa.debian.org/reproducible-builds/strip-nondeterminism https://salsa.debian.org/reproducible-builds/strip-nondeterm... There's lots of info on the Debian site about their reproducibility efforts, and there's a story from 2024's DebConf that may be of interest: https://lwn.net/Articles/985739/ https://lwn.net/Articles/985739/
  - frakkingcylons 2y ago
    I see this is written in Perl, is that the case with most Debian tooling?
    - dannyobrien 2y ago
      some, but not all. There's a bunch of historical code which means that Perl is in the base install, but modern tooling has a lot of Python too, as well as POSIX shell (not bash).
      - alfiedotwtf 2y ago
        Though a lot of the apt tooling is definitely written in Perl the last time I had to deep dive
        johnisgood 2y ago
        And a lot of OpenBSD-related stuff is written in Perl, too. I do not think it is a bad thing at all.
        alfiedotwtf 2y ago
        I absolutely love Perl. I'm just so sad Python won because Google blessed it as a language and at the time everyone wanted to work for Google. Perl always gets hate on HN, but I actually wonder of those commenter, who has actually spent over a single hours using Perl after they've read the Camel book. Honest opinion: if you're going to be spending time in Linux in your career, then you should read the Camel book at least once. Then and only then should you get to have an opinion on Perl!
- echoangle 2y ago
  Maybe dumb question but why would this change the reproducibility? If you clone a git repo, do you not get the meta data as it is stored in git? Or would the files have the modification date of the cloning? I never actually checked that.
  - mathfailure 2y ago
    You clone source from git, but then you use them to build some artifacts. The artifacts build time may differ, yet with reproducible builds - the artifact should match.
    - echoangle 2y ago
      Right, but if you only clone and build, why would the files modification date be different compared to the version that was committed to git? Does just cloning a repo already lead to different file modification dates in my local copy?
      - hoten 2y ago
        Git does not store or restore file modification times.
        codetrotter 2y ago
        And the reason for that in turn is because if you are on one commit and check out and older commit, then restoring file modification times to what they were at the time of the older commit would cause build tools that look at file modification times to sometimes not pick up on all the changes.
        echoangle 2y ago
        Ah ok, that explains it.
- paulddraper 2y ago
  > Do they forge them? Yes. All archive entries and date source code macros and any other timestamps are set to a standardized date (in the past).
  - lamby 2y ago
    This is not quite right. At least in Debian, only files that are newer than some standardised date are to that standardised date. This "clamping" preserves any metadata in older files.
- TacticalCoder 2y ago
  > ... what about files metadata like creation/modification timestamps? Do they forge them? The least difficult to solve for reproducible build but yes. The real question is: why, in the past, was an entire ecosystem created where non-determinism was the norm and everybody thought it was somehow ok? Instead of asking: "how one achieves reproducibility?" we may wonder "why did people got out of their way to make sure something as simple as a timestamp would screw determinism?". For that's the anti-security mindset we have to fight. And Debian did.
  - BobbyTables2 2y ago
    You’re forgetting that source control used to not be a mainstream practice… Software was more artisanal in nature…
  - brohee 2y ago
    TBH security is someone the source of the issues, as it often involves adding randomness. For example, replacing deterministic hashes by keyed hashes to protect from hash flooding DoS led to deterministic output becoming nondeterministic (e.g. when displaying a hash table in its natural order). Sorting had to be added to that kind of output.
c0l0 2y ago
I never really understood the hype around reproducible builds. It seems to mostly be a vehicle to enable tivoization[0] while keeping users sufficiently calm. With reproducible buiilds, a vendor can prove to users that they did build $binary from $someopensourceproject, and then digitally sign the result so that it - and only it - would load and execute on the vendor-provided and/or vendor-controlled platform. But that still kills effective software freedom as long as I, the user, cannot do the same thing with my own build (whether it is unmodified or not) of $someopensourceproject. Therefore, I side with Tavis Ormandy on this debate: https://web.archive.org/web/20210616083816/https://blog.cmpxchg8b.com/2020/07/you-dont-need-reproducible-builds.html https://web.archive.org/web/20210616083816/https://blog.cmpx... [0]: https://en.wikipedia.org/wiki/Tivoization https://en.wikipedia.org/wiki/Tivoization
- klysm 2y ago
  One of the big advantages from my perspective is you can cache a lot more effectively throughout the build process when things are deterministic.
  - c0l0 2y ago
    To achieve that it is enough to hash inputs, and cache resulting outputs. Repeating a build from scratch with an emtpy cache would not necessarily have to yield the same hashes all they way down to the last artifact, but that's actually a simplification of the whole process, and not a bad thing per se.
    - mschuster91 2y ago
      > To achieve that it is enough to hash inputs, and cache resulting outputs. Thing is, inputs can be nondeterministic too - some programs (used to) embed the current git commit hash into the final binary so that a `./foo --version` gives a quick and easy way for bug triage to check if the user isn't using a version from years ago.
      - layer8 2y ago
geocrasher 2y ago
What is the significance of a reproducible build, and how is it different than a normal distribution?
- b112 2y ago
  It means you can build it yourself, and know the source code you have, is all there is. It validates that publicly available downloads aren't different from what is claimed.
- genpfault 2y ago
  https://en.wikipedia.org/wiki/Reproducible_builds https://en.wikipedia.org/wiki/Reproducible_builds https://wiki.debian.org/ReproducibleBuilds/About https://wiki.debian.org/ReproducibleBuilds/About
- csense 2y ago
  Reproducible: If Alice and Bob both download and compile the same source code, Alice's binary is byte-for-byte identical to Bob's binary. Normal: Before Debian's initiative to handle this problem, most people didn't think hard about all the ways system-specific differences might wind up in binaries. For example: __DATE__ and __TIME__ macros in C, parallel builds finishing in different order, anything that produces a tar file (or zip etc.) usually by default asks the OS for the input files' modification time and puts that into the bytes of the tar file, filesystems may list files in a directory in different order and this may also get preserved in tar/zip files or other places... Why it's important: With reproducible builds, anyone can check the official binaries of Debian match the source code. This means going forward, any bad actors who want to sneak backdoors or other malware into Debian will have to find a way to put it in the source code, where it will be easier for people to spot.
  - walrus01 2y ago
    as the 'xz' backdoor was in the source code, and remained there for a while before anyone spotted it, it doesn't necessarily guarantee that backdoors/malware won't make their way into the source of a very-widely-redistributed project.
zozbot234 2y ago
Nice, these live images could become the foundation for a Debian-based "immutable OS" workflow.
- polynox 2y ago
  That is the goal of Vanilla OS! https://vanillaos.org/ https://vanillaos.org/
abdullahkhalids 2y ago
Is the build infrastructure for Debian also reproducible? It seems like we if someone wants to inject malware in Debian package binaries (without injecting them into the source), they have to target the build infrastructure (compilers, linkers and whatever wrapper code is written around them). Also, is someone else also compiling these images, so we have evidence that the Debian compiling servers were not compromised?
- jzb 2y ago
  There's a page that includes reproducibility results for Debian here: https://tests.reproducible-builds.org/debian/bookworm/index_suite_amd64_stats.html https://tests.reproducible-builds.org/debian/bookworm/index_... I think there's also a similar thing for the images, but I might be wrong and I definitely don't have the link handy at the moment. There's lots of documentation about all of the things on Debian's site at the links in the brief. And LWN also had a story last year about Holger Levsen's talk on the topic from DebConf: https://lwn.net/Articles/985739/ https://lwn.net/Articles/985739/
- deleted 2y ago
  [deleted]
- layer8 2y ago
  And what about the hardware on which the build runs? Is it reproducible? ;)
  - nikisweeting 2y ago
    well little johnny, when one hardware loves another hardware very much...
  - kragen 2y ago
    Working on it! But in general the answer is that for most purposes it's good enough to show that many independently produced pieces of hardware can reproduce the same results.
  - ratmice 2y ago
jcmfernandes 2y ago
Insane effort. This sounded like a pipe dream just a couple of years ago. Congrats to everyone involved, especially to those who drove the effort.
- Joel_Mckay 2y ago
  The Debian group is admirable, and have positively changed the standards for OS design several times. Reminds me I should donate to their coffee fund around tax time =3
  - alfiedotwtf 2y ago
    Exactly! I’ve said it many times and I’ll repeat it here - Debian will be one of the few Linux distros we have right now, that will still exist 100 years from now. Yea, it’s not as modern in terms of versioning and risk compared to the likes of Arch, but that’s also a feature!
    - walrus01 2y ago
      It's quite easy to run Debian unstable (sid) if you want a more risky approach to having the newest of every package.
      - cess11 2y ago
        Commonly these days you can also add specific repos for the things you want to be more on the edge. Then there are some tools one might install manually, at the moment I remember doing it with fzf.
        sgarland 2y ago
        Yep, I do this for a few tools. Though apt-key deprecation still hasn’t been universally accepted, so that’s always a minor annoyance to deal with.
        presbyterian 2y ago
kragen 2y ago
This is a huge milestone: https://lists.reproducible-builds.org/pipermail/rb-general/2025-March/003675.html https://lists.reproducible-builds.org/pipermail/rb-general/2...
Cort3z 2y ago
I’m a noob to this subject. How can a build be non-reproducible? By that, I mean, what part of the build process could return non-deterministic output? Are people putting timestamps into the build and stuff like that?
- r3trohack3r 2y ago
  File paths, timestamps, unstable ordering of inputs/outputs, locals, version info, variations in the build environment, etc. This pages has a good write up https://reproducible-builds.org/docs/ https://reproducible-builds.org/docs/
- jcranmer 2y ago
  Timestamps, timestamps, absolute paths (i.e., differences between building /src versus /home/Cort3z/source), timestamps, file inode numbering ("for file in directory" defaults to inode order rather than alphabetical order in many languages, and that means it's effectively pseudorandom), more timestamps, using random data in your build process (e.g., embedding a generated private key, or signing something), timestamps, and accidental nondeterminism within the compiler. By far the most prevalent source of nondeterminism is timestamps, especially since timestamps crop up in file formats you don't expect (e.g., running gzip stuffs a timestamp in its output for who knows what reason). After that, it's the two big filesystem issues (absolute paths and directory iteration nondeterminism), and then it's basically a long tail of individual issues that affect but one or two packages.
perdomon 2y ago
Can someone please ELI5? When I hear live images, I think of iOS videos that go along with pictures you take
- mjg59 2y ago
  Live images are Linux distributions that can be run directly from removable media instead of having to be installed to local storage.
  - nottorp 2y ago
    And iOS live images are half second movies, not images.
- stuporglue 2y ago
  A live image is an operating system image which you can boot from and use vs. an install disk which can only install, but there's no usable environment available). A reproducable build means you can get the same source code and compile it, and it will be identical to the published image. This is important because otherwise you don't know if the published image actually used some other source code. If it used some other source code, the published image might have a backdoor, or something that you can't find by reading the source code.
  - perdomon 2y ago
    Is that the idea behind Tails OS? It runs from removable media and disappears when ejected?
    - abdullahkhalids 2y ago
      Yes. Though the disappearing doesn't happen when you eject the removable media. When you first boot from the removable media, the OS loads itself into the RAM. If you want to open additional programs, then those are loaded from the media into RAM and then executed. However, you can remove the media at any point after boot, and after that you only run the programs that are already loaded into RAM. Also we have had live images of various OSes for many decades. I seem to recall that we used to load DOS from floppy disks.
yupyupyups 2y ago
This is amazing news. Well done!
moondev 2y ago
Do these live images come ready with cloud-init? A cloud-init in-memory live iso seems perfect for immutable infrastructure "anywhere"
- bravetraveler 2y ago
  Should be trivial to put in, if not. Install the package and maybe prepare some datasource hints while reproducing the image. Depends on where you'll be using it. The trick will be in the details, as usual. User data that both does useful work... and plays nicely with immutability. I suspect it would be more sensible to skip the gymnastics of trying to manicure something inherently resistant, and instead, lean in on reproducibility. Make it as you want it, skip the extra work. Want another? Great - they're freely reproducible :)
selfhoster 2y ago
[flagged]
curtisszmania 2y ago
Pretty wild that we’re finally nailing reproducibility in Linux images after so many years—clearly a win for stability and consistency across the board.
letters90 2y ago
the update is gold, original message: "They are reproduceable" updated message "lol actually not"
- eqvinox 2y ago
  Not really, it's just someone with a higher goal of "freedom" (no binary firmware blobs) using it to push their agenda. I'll happily agree higher degrees of "freedom" are an admirable goal, but this is just rudely shitting on a hard-earned achievement.
nwellinghoff 2y ago
Does anyone have any information as to how they modified their C code such that the complier output was deterministic? I thought one of the hardest problems with a effort like this was writing your C such that the compiler would output everything in the same order (same bytes)? And I am not just talking about time stamps etc.
- account42 2y ago
  This is something compilers themselves need to guarantee, e.g. GCC has -frandom-seed [0]. [0] https://gcc.gnu.org/onlinedocs/gcc/Developer-Options.html#index-frandom-seed https://gcc.gnu.org/onlinedocs/gcc/Developer-Options.html#in...
amelius 2y ago
How does that work with timestamps?
kroeckx 2y ago
It's my understanding that is about generating the .iso file from the .deb files, not about generating the .deb files from source. Generating .deb from source in a reproducible way is still a work in progress.
hackburg 2y ago
[dead]

766 points by bertman 2y ago

I don't get how someone achieves reproducibility of builds: what about files metadata like creation/modification timestamps? Do they forge them? Or are these data treated as not important enough (like it 2 files with different metadata but identical contents should have the same checksum when hashed)?

HideousKojima 2y ago
Those aren't needed to generate a hash of a file. And that metadata isn't part of the file itself (or at least doesn't need to be), it's part of the filesystem or OS
- imcritic 2y ago
  That's an acceptable answer for the simple case when you distribute just a file, but what if your distribution is something more complex, like an archive with some sub-archives? Metadata in the internal files will affect the checksum of the resulting archive.
  - exe34 2y ago
    unless you fix them to a known epoch.
  - londons_explore 2y ago
    Finding and fixing cases like this are part of what the project has done...
c0l0 2y ago
Yes.
o11c 2y ago
Timestamps are easiest part - you just set everything according to the chosen epoch. The hard things involve things like unstable hash orderings, non-sorted filesystem listing, parallel execution, address-space randomization, ...
- koolba 2y ago
  ASLR shouldn’t be an issue unless you intend to capture the entire memory state of the application. It’s an intermediate representation in memory, not an output of any given step of a build. Annoying edge cases come up for things like internal object serialization to sort things like JSON keys in config files.
  - sodality2 2y ago
    Let’s say a compiler is doing something in a multi-threaded manner - isn’t it possible that ASLR would affect the ordering of certain events which could change the compiled output? Sure you could just set threads to 1 but there’s probably some more edge cases in there I haven’t thought of.
    - zamadatix 2y ago
      I think you'd need the compiler to guarantee serialization order of such operations regardless if you used ASLR or not. Otherwise you're just hoping thread scheduling, core clocking, thread memory access, and many other things are the same between every system trying to do a reproducible build. Even setting threads to 1 may not solve that problem class if asynchronous functions/syscalls come into play.
  - cperciva 2y ago
    FreeBSD tripped over an issue recently where a C++ program (I think clang?) used a collection of pointers and output values in an order based on the pointers rather than the values they pointed to. ASLR by itself shouldn't cause reproducibility issues, but it can certainly expose bugs.
    - 2y ago
purkka 2y ago
Generally, yes: https://reproducible-builds.org/docs/timestamps/ https://reproducible-builds.org/docs/timestamps/ Since the build is reproducible, it should not matter when it was built. If you want to trace a build back to its source, there are much better ways than a timestamp.
- ryandrake 2y ago
  C compilers offer __DATE__ and __TIME__ macros, which expand to string constants that describe the date and time that the preprocessor was invoked. Any code using these would have different strings each time it was built, and would need to be modified. I can't think of a good reason for them to be used in an actual production program, but for whatever reason, they exist.
  - fmbb 2y ago
    Toolchains for reproducible software likely let you set these values, or ensure they are 1970-01-01 00:00:00
    - mikepurvis 2y ago
      Nix sets everything to the epoch, although I believe Debian's approach is to just use the date of the newest file in the dsc tarballs.
      - yjftsjthsd-h 2y ago
        Nix can also set it to things other than 0; I think my favorite is to set it by the time of the commit from which you're building.
        deleted 2y ago
        [deleted]
        terinjokes 2y ago
        Which is also used when the contents of a derivation will be included in a zip file. The Unix epoch is about a decade older than the zip epoch.
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
jzb 2y ago
Debian uses a tool called `strip-nondeterminism` to help with this in part: https://salsa.debian.org/reproducible-builds/strip-nondeterminism https://salsa.debian.org/reproducible-builds/strip-nondeterm... There's lots of info on the Debian site about their reproducibility efforts, and there's a story from 2024's DebConf that may be of interest: https://lwn.net/Articles/985739/ https://lwn.net/Articles/985739/
- frakkingcylons 2y ago
  I see this is written in Perl, is that the case with most Debian tooling?
  - dannyobrien 2y ago
    some, but not all. There's a bunch of historical code which means that Perl is in the base install, but modern tooling has a lot of Python too, as well as POSIX shell (not bash).
    - alfiedotwtf 2y ago
      Though a lot of the apt tooling is definitely written in Perl the last time I had to deep dive
      - johnisgood 2y ago
        And a lot of OpenBSD-related stuff is written in Perl, too. I do not think it is a bad thing at all.
        alfiedotwtf 2y ago
        I absolutely love Perl. I'm just so sad Python won because Google blessed it as a language and at the time everyone wanted to work for Google. Perl always gets hate on HN, but I actually wonder of those commenter, who has actually spent over a single hours using Perl after they've read the Camel book. Honest opinion: if you're going to be spending time in Linux in your career, then you should read the Camel book at least once. Then and only then should you get to have an opinion on Perl!
echoangle 2y ago
Maybe dumb question but why would this change the reproducibility? If you clone a git repo, do you not get the meta data as it is stored in git? Or would the files have the modification date of the cloning? I never actually checked that.
- mathfailure 2y ago
  You clone source from git, but then you use them to build some artifacts. The artifacts build time may differ, yet with reproducible builds - the artifact should match.
  - echoangle 2y ago
    Right, but if you only clone and build, why would the files modification date be different compared to the version that was committed to git? Does just cloning a repo already lead to different file modification dates in my local copy?
    - hoten 2y ago
      Git does not store or restore file modification times.
      - codetrotter 2y ago
        And the reason for that in turn is because if you are on one commit and check out and older commit, then restoring file modification times to what they were at the time of the older commit would cause build tools that look at file modification times to sometimes not pick up on all the changes.
      - echoangle 2y ago
        Ah ok, that explains it.
paulddraper 2y ago
> Do they forge them? Yes. All archive entries and date source code macros and any other timestamps are set to a standardized date (in the past).
- lamby 2y ago
  This is not quite right. At least in Debian, only files that are newer than some standardised date are to that standardised date. This "clamping" preserves any metadata in older files.
TacticalCoder 2y ago
> ... what about files metadata like creation/modification timestamps? Do they forge them? The least difficult to solve for reproducible build but yes. The real question is: why, in the past, was an entire ecosystem created where non-determinism was the norm and everybody thought it was somehow ok? Instead of asking: "how one achieves reproducibility?" we may wonder "why did people got out of their way to make sure something as simple as a timestamp would screw determinism?". For that's the anti-security mindset we have to fight. And Debian did.
- BobbyTables2 2y ago
  You’re forgetting that source control used to not be a mainstream practice… Software was more artisanal in nature…
- brohee 2y ago
  TBH security is someone the source of the issues, as it often involves adding randomness. For example, replacing deterministic hashes by keyed hashes to protect from hash flooding DoS led to deterministic output becoming nondeterministic (e.g. when displaying a hash table in its natural order). Sorting had to be added to that kind of output.

c0l0 2y ago

I never really understood the hype around reproducible builds. It seems to mostly be a vehicle to enable tivoization[0] while keeping users sufficiently calm. With reproducible buiilds, a vendor can prove to users that they did build $binary from $someopensourceproject, and then digitally sign the result so that it - and only it - would load and execute on the vendor-provided and/or vendor-controlled platform. But that still kills effective software freedom as long as I, the user, cannot do the same thing with my own build (whether it is unmodified or not) of $someopensourceproject. Therefore, I side with Tavis Ormandy on this debate: https://web.archive.org/web/20210616083816/https://blog.cmpxchg8b.com/2020/07/you-dont-need-reproducible-builds.html https://web.archive.org/web/20210616083816/https://blog.cmpx... [0]: https://en.wikipedia.org/wiki/Tivoization https://en.wikipedia.org/wiki/Tivoization

klysm 2y ago
One of the big advantages from my perspective is you can cache a lot more effectively throughout the build process when things are deterministic.
- c0l0 2y ago
  To achieve that it is enough to hash inputs, and cache resulting outputs. Repeating a build from scratch with an emtpy cache would not necessarily have to yield the same hashes all they way down to the last artifact, but that's actually a simplification of the whole process, and not a bad thing per se.
  - mschuster91 2y ago
    > To achieve that it is enough to hash inputs, and cache resulting outputs. Thing is, inputs can be nondeterministic too - some programs (used to) embed the current git commit hash into the final binary so that a `./foo --version` gives a quick and easy way for bug triage to check if the user isn't using a version from years ago.
    - layer8 2y ago

geocrasher 2y ago

What is the significance of a reproducible build, and how is it different than a normal distribution?

b112 2y ago
It means you can build it yourself, and know the source code you have, is all there is. It validates that publicly available downloads aren't different from what is claimed.
genpfault 2y ago
https://en.wikipedia.org/wiki/Reproducible_builds https://en.wikipedia.org/wiki/Reproducible_builds https://wiki.debian.org/ReproducibleBuilds/About https://wiki.debian.org/ReproducibleBuilds/About
csense 2y ago
Reproducible: If Alice and Bob both download and compile the same source code, Alice's binary is byte-for-byte identical to Bob's binary. Normal: Before Debian's initiative to handle this problem, most people didn't think hard about all the ways system-specific differences might wind up in binaries. For example: __DATE__ and __TIME__ macros in C, parallel builds finishing in different order, anything that produces a tar file (or zip etc.) usually by default asks the OS for the input files' modification time and puts that into the bytes of the tar file, filesystems may list files in a directory in different order and this may also get preserved in tar/zip files or other places... Why it's important: With reproducible builds, anyone can check the official binaries of Debian match the source code. This means going forward, any bad actors who want to sneak backdoors or other malware into Debian will have to find a way to put it in the source code, where it will be easier for people to spot.
- walrus01 2y ago
  as the 'xz' backdoor was in the source code, and remained there for a while before anyone spotted it, it doesn't necessarily guarantee that backdoors/malware won't make their way into the source of a very-widely-redistributed project.

zozbot234 2y ago

Nice, these live images could become the foundation for a Debian-based "immutable OS" workflow.

polynox 2y ago
That is the goal of Vanilla OS! https://vanillaos.org/ https://vanillaos.org/

abdullahkhalids 2y ago

Is the build infrastructure for Debian also reproducible? It seems like we if someone wants to inject malware in Debian package binaries (without injecting them into the source), they have to target the build infrastructure (compilers, linkers and whatever wrapper code is written around them). Also, is someone else also compiling these images, so we have evidence that the Debian compiling servers were not compromised?

jzb 2y ago
There's a page that includes reproducibility results for Debian here: https://tests.reproducible-builds.org/debian/bookworm/index_suite_amd64_stats.html https://tests.reproducible-builds.org/debian/bookworm/index_... I think there's also a similar thing for the images, but I might be wrong and I definitely don't have the link handy at the moment. There's lots of documentation about all of the things on Debian's site at the links in the brief. And LWN also had a story last year about Holger Levsen's talk on the topic from DebConf: https://lwn.net/Articles/985739/ https://lwn.net/Articles/985739/
deleted 2y ago
[deleted]
layer8 2y ago
And what about the hardware on which the build runs? Is it reproducible? ;)
- nikisweeting 2y ago
  well little johnny, when one hardware loves another hardware very much...
- kragen 2y ago
  Working on it! But in general the answer is that for most purposes it's good enough to show that many independently produced pieces of hardware can reproduce the same results.
- ratmice 2y ago

jcmfernandes 2y ago

Insane effort. This sounded like a pipe dream just a couple of years ago. Congrats to everyone involved, especially to those who drove the effort.

Joel_Mckay 2y ago
The Debian group is admirable, and have positively changed the standards for OS design several times. Reminds me I should donate to their coffee fund around tax time =3
- alfiedotwtf 2y ago
  Exactly! I’ve said it many times and I’ll repeat it here - Debian will be one of the few Linux distros we have right now, that will still exist 100 years from now. Yea, it’s not as modern in terms of versioning and risk compared to the likes of Arch, but that’s also a feature!
  - walrus01 2y ago
    It's quite easy to run Debian unstable (sid) if you want a more risky approach to having the newest of every package.
    - cess11 2y ago
      Commonly these days you can also add specific repos for the things you want to be more on the edge. Then there are some tools one might install manually, at the moment I remember doing it with fzf.
      - sgarland 2y ago
        Yep, I do this for a few tools. Though apt-key deprecation still hasn’t been universally accepted, so that’s always a minor annoyance to deal with.
      - presbyterian 2y ago

kragen 2y ago

This is a huge milestone: https://lists.reproducible-builds.org/pipermail/rb-general/2025-March/003675.html https://lists.reproducible-builds.org/pipermail/rb-general/2...

Cort3z 2y ago

I’m a noob to this subject. How can a build be non-reproducible? By that, I mean, what part of the build process could return non-deterministic output? Are people putting timestamps into the build and stuff like that?

r3trohack3r 2y ago
File paths, timestamps, unstable ordering of inputs/outputs, locals, version info, variations in the build environment, etc. This pages has a good write up https://reproducible-builds.org/docs/ https://reproducible-builds.org/docs/
jcranmer 2y ago
Timestamps, timestamps, absolute paths (i.e., differences between building /src versus /home/Cort3z/source), timestamps, file inode numbering ("for file in directory" defaults to inode order rather than alphabetical order in many languages, and that means it's effectively pseudorandom), more timestamps, using random data in your build process (e.g., embedding a generated private key, or signing something), timestamps, and accidental nondeterminism within the compiler. By far the most prevalent source of nondeterminism is timestamps, especially since timestamps crop up in file formats you don't expect (e.g., running gzip stuffs a timestamp in its output for who knows what reason). After that, it's the two big filesystem issues (absolute paths and directory iteration nondeterminism), and then it's basically a long tail of individual issues that affect but one or two packages.

perdomon 2y ago

Can someone please ELI5? When I hear live images, I think of iOS videos that go along with pictures you take

mjg59 2y ago
Live images are Linux distributions that can be run directly from removable media instead of having to be installed to local storage.
- nottorp 2y ago
  And iOS live images are half second movies, not images.
stuporglue 2y ago
A live image is an operating system image which you can boot from and use vs. an install disk which can only install, but there's no usable environment available). A reproducable build means you can get the same source code and compile it, and it will be identical to the published image. This is important because otherwise you don't know if the published image actually used some other source code. If it used some other source code, the published image might have a backdoor, or something that you can't find by reading the source code.
- perdomon 2y ago
  Is that the idea behind Tails OS? It runs from removable media and disappears when ejected?
  - abdullahkhalids 2y ago
    Yes. Though the disappearing doesn't happen when you eject the removable media. When you first boot from the removable media, the OS loads itself into the RAM. If you want to open additional programs, then those are loaded from the media into RAM and then executed. However, you can remove the media at any point after boot, and after that you only run the programs that are already loaded into RAM. Also we have had live images of various OSes for many decades. I seem to recall that we used to load DOS from floppy disks.

yupyupyups 2y ago

This is amazing news. Well done!

moondev 2y ago

Do these live images come ready with cloud-init? A cloud-init in-memory live iso seems perfect for immutable infrastructure "anywhere"

bravetraveler 2y ago
Should be trivial to put in, if not. Install the package and maybe prepare some datasource hints while reproducing the image. Depends on where you'll be using it. The trick will be in the details, as usual. User data that both does useful work... and plays nicely with immutability. I suspect it would be more sensible to skip the gymnastics of trying to manicure something inherently resistant, and instead, lean in on reproducibility. Make it as you want it, skip the extra work. Want another? Great - they're freely reproducible :)

selfhoster 2y ago

[flagged]

curtisszmania 2y ago

Pretty wild that we’re finally nailing reproducibility in Linux images after so many years—clearly a win for stability and consistency across the board.

letters90 2y ago

the update is gold, original message: "They are reproduceable" updated message "lol actually not"

eqvinox 2y ago
Not really, it's just someone with a higher goal of "freedom" (no binary firmware blobs) using it to push their agenda. I'll happily agree higher degrees of "freedom" are an admirable goal, but this is just rudely shitting on a hard-earned achievement.

nwellinghoff 2y ago

Does anyone have any information as to how they modified their C code such that the complier output was deterministic? I thought one of the hardest problems with a effort like this was writing your C such that the compiler would output everything in the same order (same bytes)? And I am not just talking about time stamps etc.

account42 2y ago
This is something compilers themselves need to guarantee, e.g. GCC has -frandom-seed [0]. [0] https://gcc.gnu.org/onlinedocs/gcc/Developer-Options.html#index-frandom-seed https://gcc.gnu.org/onlinedocs/gcc/Developer-Options.html#in...

amelius 2y ago

How does that work with timestamps?

kroeckx 2y ago

It's my understanding that is about generating the .iso file from the .deb files, not about generating the .deb files from source. Generating .deb from source in a reproducible way is still a work in progress.

hackburg 2y ago

[dead]