18 ms·

NixOS Reproducible Builds: minimal ISO successfully independently rebuilt

548 points by CathalMullan 3y ago

deleted 3y ago
[deleted]
mihalycsaba 3y ago
Sorry for being dense, but I thought one of the main reason for nixos's existence is reproducibilty. I thought they have these kinds of things solved already. I have only ~2 hours experience with Nixos, wanted to try hyprland, I thought it would be easier on Nixos since hyprland needs a bit of setup and maybe it's easier to use someone else's config on nixos, than on some other distro. Finding a config was hard too, found like 3 on some random github gists, thought there would be more... and none of them worked, at that point I gave up.
- rgoulter 3y ago
  Yeah, Nix is a tough tool to learn. It's probably never the right tool to pick for "I just want something that works right now" if you're unfamiliar with it. > I thought one of the main reason for nixos's existence is reproducibilty NixOS uses "reproducible" to mean "with the same Nix code, you get the same program behaviour". This is more/less what people hope Dockerfiles provide. This is the level of reproducibility you want when you say "it works on my machine" or "it worked last time I tried it". Whereas "reproducible build" aims for bit-for-bit equality for artifacts build on different machines. -- With this, there's a layer of security in that you can verify that code has been built from a particular set of sources. > Finding a config was hard too What search query were you using? Searching "nixos configuration" on https://github.com/search?q=nixos%20configuration&type=repositories https://github.com/search?q=nixos%20configuration&type=repos... Or searching for hyprland specifically, there seem to be many using that https://github.com/search?q=wayland.windowManager.hyprland&type=code https://github.com/search?q=wayland.windowManager.hyprland&t...
  - amarshall 3y ago
    > NixOS uses "reproducible" to mean "with the same Nix code, you get the same program behaviour". Note that ”Nix code” also includes the hashes of all non-Nix sources. One way to think of it is that Nix has reliable build cache invalidation. > This is more/less what people hope Dockerfiles provide. Indeed, but importantly they do not provide input-reproducibility (while Nix does) because, at least, there are no hashes for remote data.
    - __MatrixMan__ 3y ago
      PSA: you can build OCI images with nix, then they'll be a pure function of their input like we've all wished was the case with Dockerfiles.
      - Smaug123 3y ago
        (And Nix derivations compose, whereas Dockerfiles entirely do not.)
  - mihalycsaba 3y ago
    I don't remember, some of them needed some other tools installed(like flakes whatever it is), I looked for configs, that looked like they don't need a few more hours to learn and to setup some other tools for them to work. I just wanted to take a quick look at hyprland, I imagined I just use an existing config, I never thought it would need hours of research. Later I installed an arch vm and managed to install hyprland with some basic components in less than an hour from the first guide I found. Looks like I misunderstood, what nix was made for. I just want a system I can more or less set up with a simple config file. I saw this os, didn't have time to try it yet, but I thought this is how nix works. https://blendos.co/ https://blendos.co/ For example you just define gnome like this, the nix configs I found looked similar, they just didn't work. >gnome: > enabled: true > style: light > gtk-theme: 'adw-gtk3' > icon-theme: 'Adwaita' > titlebar: > button-placement: 'right' > double-click-action: 'toggle-maximize' > middle-click-action: 'minimize' > right-click-action: 'menu'
- quietbritishjim 3y ago
  There are two senses of reproducible. The sense you're thinking of is that you can easily rebuild a binary package and it will use the same dependency versions, build options, etc. There should be no chance of a compiler error that didn't happen the first time (the old "but it worked on my laptop" syndrome). The sense used here is that every build output is byte-for-byte binary identical. It doesn't depend on the machine name, the time it was compiled or anything like that (or, in a parallel build, the order in which files finish compiling). That is much harder.
  - jowea 3y ago
    > The sense you're thinking of is that you can easily rebuild a binary package and it will use the same dependency versions, build options, etc. There should be no chance of a compiler error that didn't happen the first time (the old "but it worked on my laptop" syndrome). And that's just for Nixpkgs, the packages themselves that also work outside NixOS. NixOS has reproducibility of the entire system complete with configuration.
- ParetoOptimal 3y ago
  > Finding a config was hard too, found like 3 on some random github gists, thought there would be more.. That sounds odd, did you use github code search? Find relevant home manager options: https://mipmip.github.io/home-manager-option-search/?query=hyprland https://mipmip.github.io/home-manager-option-search/?query=h... Then search those on github: https://github.com/search?utf8=%E2%9C%93&q=lang%3Anix+hyprland+plugins&type=code https://github.com/search?utf8=%E2%9C%93&q=lang%3Anix+hyprla... Note some option searches imply more casual or advanced users.
- chpatrick 3y ago
  > Sorry for being dense, but I thought one of the main reason for nixos's existence is reproducibilty. I thought they have these kinds of things solved already. Nixos has the advantage that everything is built in its own sandbox with only its explicitly declared (and hashed) dependencies available, unlike in mainstream distros where it's the full system environment, so in many cases you already get the same binary every time. But this doesn't immediately lead to reproducibility because the build process might be nondeterministic for various packages.
  - benreesman 3y ago
    This is a really good comment, I have no idea why it’s going grey. Upvote from me FWIW.
  - WhyNotHugo 3y ago
    > unlike in mainstream distros where it's the full system environment Usually packages are built in an environment which has only a minimal base system plus the package's explicitly dependencies. They don't have random unnecessary packages installed.
  - goodpoint 3y ago
    > unlike in mainstream distros Debian has been building in a clean sandbox with only required, tracked dependencies since decades. It's also building the large majority of packages reproducibly including the binary and whole installation packages (not just the sources like nixos)
    - chpatrick 3y ago
      > not just the sources like nixos Not sure what you mean by that, the Nix packages that are reproducible have reproducible binaries. In the Nixos world there isn't really a concept of a "binary/installation package" like in Debian or elsewhere. Everything can be rebuilt from source on any machine, but because everything is hashed, if the official binary caches have already built something with the same inputs, they can just give you the outputs directly. So it's more like memoization than a .deb or something that you install. Nix is a functional language that builds recipes (derivations) to build stuff, with all the inputs and outputs hashed. If the derivation you want to build has already been built by a cache you trust, the system will just fetch it instead of building locally. What the Nix reproducability project checks is that the same derivation produces the same output regardless of what machine it's built on.
- Aerbil313 3y ago
  Check out https://github.com/donovanglover/nix-config https://github.com/donovanglover/nix-config . Flake based config with hyprland and cool stuff. > at that point I gave up. NixOS is not for the weak or time constrained, currently. Hopefully it will be one day. Still if you push through, you reap the benefits.
  - flkiwi 3y ago
    Another good option: https://github.com/Misterio77/nix-starter-configs https://github.com/Misterio77/nix-starter-configs I started with this one, the minimal version, then moved on to something more like the standard version, and now I'm moving on to something based on his much more complicated and flexible build in a different repo. I had been flailing, then this repo made it click.
    - __MatrixMan__ 3y ago
      Same here, that repo is fantastic.
      - flkiwi 3y ago
        His documentation has gotten better too! I actually just rebuilt my entire config based on his updated "standard" version. I want to use his "non-starter" config too, bc it seems remarkably powerful, but ... I need more time and brain to do that. And, just to be clear, I rebuilt my entire config in less than an hour, copying and pasting where necessary, then it took about 20 minutes (!!) to download and setup all my software. Only one error, so I consider that a significant My NixOS Journey accomplishment! Granted, because it's NixOS, I have NO IDEA AT ALL what the error was, lol.
- colordrops 3y ago
  Nix is reproducteable in tbe environment sense, meaning you can get the exact same setup every time, but not in the bit-for-bit sense, meaning that the compiled binaries will be identical.
Reventlov 3y ago
For those wondering : it should be remembered that the reproducibility of Nix / NixOS / Nixpkgs is only a reproducibility of the sources: if the sources change, one is warned, but it is not a question of the reproducibility of the binaries (which can change at each build). This binary reproducibility of Nix / NixOS / Nixpkgs is indeed not really tested, at least not systematically. Guix, Archlinux, Debian do the binary reproducibility better than Nix / NixOS / Nixpkgs. Sources : - https://r13y.com/ https://r13y.com/ ( Nix* ) - https://tests.reproducible-builds.org/debian/reproducible.html https://tests.reproducible-builds.org/debian/reproducible.ht... ( Debian ) - https://tests.reproducible-builds.org/archlinux/archlinux.html https://tests.reproducible-builds.org/archlinux/archlinux.ht... ( Archlinux ) - https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/package-reproducibility https://data.guix.gnu.org/repository/1/branch/master/latest-... (Guix, might be a bit slow to load, here is some cached copy https://archive.is/lTuPk https://archive.is/lTuPk )
- dicytea 3y ago
  > Guix, Archlinux, Debian do the binary reproducibility better than Nix / NixOS / Nixpkgs. Huh, didn't know that Arch Linux tests reproducibility. It's apparently 85.6% reproducible: https://reproducible.archlinux.org https://reproducible.archlinux.org I wonder how much work would be needed for NixOS, considering it has more than 80k packages in the official repository.
  - chpatrick 3y ago
    I think that's also a bit of an unfair comparison given the number of AUR packages you usually use on Arch. With nixpkgs there isn't a distinction between official and community packages.
    - deleted 3y ago
      [deleted]
    - iopq 3y ago
      Sure there is, the NUR has a few thousand community packages that are not ready for release The nixpkgs are all official packages, it's just really easy to become a maintainer (you make a pull request adding the package you want to maintain)
onedognight 3y ago
Rebuilding the minimal ISO from source is an impressive milestone on the journey to a system that builds from source reproducibly. Guix had an orthogonal but equally impressive milestone on the same journey recently[0], bootstrapping a full compiler toolchain from a single reproducible 357 byte binary without any other binary compiler blobs. These two features may one day soon be combined to reproducibly build a full distribution from source. [0] https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-building-from-source-all-the-way-down/ https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-...
- Thrir94994i 3y ago
  357 bytes for bootstrap compiler binary is VERY impressive!
  - msm_ 3y ago
    If I remember correctly, this tiny binary is used to (reproducibly) bootstrap the next binary, which bootstraps the next binary, until eventually GCC can be compiled (and compile other software).
    - Smaug123 3y ago
      The bootstrap chain is https://github.com/oriansj/stage0-posix-x86/blob/e86bf7d304bae5ce5ccc88454bb60cf0837e941f/mescc-tools-mini-kaem.kaem https://github.com/oriansj/stage0-posix-x86/blob/e86bf7d304b...
  - rssoconnor 3y ago
    To be fair, it is 357 bytes ... plus a POSIX operating system. Still, that POSIX operating system bit is also being worked on.
    - 6581 3y ago
      Isn't that what builder-hex0 does? https://github.com/ironmeld/builder-hex0 https://github.com/ironmeld/builder-hex0
KennyFromIT 3y ago
I've lived in the Red Hat ecosystem for work recently. How does this compare to something like... Fedora Silverblue? Ansible? Fedora Silverblue + Ansible?
- TheDong 3y ago
  The closest equivalent to the nixos ISO builder and reproducibility related to it in the fedora ecosystem is osbuild / imagebuilder - https://www.osbuild.org/guides/introduction.html https://www.osbuild.org/guides/introduction.html Imagebuilder claims reproducibility, but as far as I know it mostly installed rpm packages as binaries, not from source, so it's not really proper reproducibility unless all the input packages are also reproducible. If the descriptions of building packages from source, building distro images, and reproducibility in the linked thread didn't make sense to you, you're probably not really the target audience anyway.
- candiddevmike 3y ago
  Nix is a declarative OS, where you describe what the OS should look like, instead of Ansible where you give the OS steps to follow. Silverblue and Nix are orthogonal aside from being Linux distributions--Silverblue is attempting to change how software is delivered using only containers on an immutable host. If you're interested in an Ansible alternative that uses Jsonnet and state tracking to somewhat mimic Nix, check out Etcha: https://etcha.dev https://etcha.dev
  - deleted 3y ago
    [deleted]
  - rgoulter 3y ago
    > Nix is a declarative OS I think precision is important. "Nix" refers to the package manager (and the language the package manager uses). Whereas it's "NixOS" that's the OS which makes use of Nix to manage the system configuration.
    - Ericson2314 3y ago
somat 3y ago
I find it funny(ironic) that the OpenBSD project is trying hard to go the other way, every single install has unique and randomized address offsets. While I understand that these two goals, reproducible builds and unique installs, are orthogonal to each other, both can be had at the same time, the duality of the situation still makes me laugh.
- oever 3y ago
  If the address offsets can be randomized with a provided seed, then demonstrating reproducibility is still possible. Alternatively, randomizing the offsets when starting the program is another way to keep reproducibility and even increase security; the offsets would change at every run.
- WhyNotHugo 3y ago
  OpenBSD does randomised linking at boot time. Packages themselves can still be reproducible. All the randomisation is done locally after the packages are downloaded and their checksums validated.
mbakke 3y ago
Very impressive milestone, congrats to those who made this possible! > [...] actually rebuilding the ISO still introduced differences. This was due to some remaining problems in the hydra cache and the way the ISO was created. Can anyone shed some light on the fix for "how the ISO was created"? I attempted making a reproducible ISO a while back but could not make the file system create extents in a deterministic fashion.
- raboof 3y ago
  For NixOS, it's in the 'how did we reproduce' section of the article: the last step of that process produces the iso in the ./result/iso directory. It sounds like what you're looking for is the commands that that build invoked, but I'm not sure what step you're looking for. For example, the xorriso invocations are at https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/make-iso9660-image.sh#L97 https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/make-...
ahmedfromtunis 3y ago
Stupid question as I never worked on something like this before: why isn't reproducibility the default behavior? I mean if 2 copies of a piece of software were compiled from the same source, what stops them from being identical each and every time? I know there are so many moving parts, but I still can't understand how discrepancies can manifest themselves.
- bravetraveler 3y ago
  I don't develop enough to give a particularly good answer, but one example I've heard of involves timestamps Imagine the program uses the current date or time as a value. When compiled at different moments, the bits change. Same applies to anything where the build environment or timing influences the output binary
- kaba0 3y ago
  Parallelism. There might be actions that are not order-independent, and the state of the CPU might result in slightly different binaries, but all are correct.
  - edgyquant 3y ago
    Why does this matter though? Why does order of compilation result in a different binary?
    - speed_spread 3y ago
      Because order of completion of the parallel tasks is not guaranteed, if all tasks write to the same file you might get a different result each time.
    - kaba0 3y ago
      Just some random, made up example: say you want to compile an OOP PL that has interfaces and implementations of that. You discover reachable implementations through static analysis, which is multi-threaded. You might discover implementations A,B,C in any order — but they will get their methods placed in the jump table based on this order. This will trivially result in semantically equivalent, but not binary-equivalent executables. Of course there would have been better designs for this toy example, but binary reproducibility is/was usually not of the highest priority historically in most compiler infrastructures, and in some cases it might be a relatively big performance regression to fix, or simply just a too big refactor.
mgaunard 3y ago
Don't you have to fake the system time to do this? The time often ends up inside the binaries one way or another.
- mbakke 3y ago
  Indeed time stamps are probably the most common sources of indeterminism. So common that a de-facto standard variable to fake a timestamp has been implemented in many compilers: https://reproducible-builds.org/docs/source-date-epoch/ https://reproducible-builds.org/docs/source-date-epoch/
- amelius 3y ago
  Could you name an example of how (and for what reason) this might happen?
  - mbakke 3y ago
    Typically part of a "version string": $ python3 Python 3.10.7 (main, Jan 1 1970, 00:00:01) [GCC 11.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> Perhaps a relic from when software had to be manually updated?
    - oever 3y ago
      On NixOS, I think the release time or commit time is used: $ python3 Python 3.10.11 (main, Apr 4 2023, 22:10:32) [GCC 12.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> That is more useful than the build time.
      - mbakke 3y ago
        How is that possible? Is nixpkgs an input to the Python derivation? Or do packagers "hard code" a value every time they modify the Python build code? Automated tooling that sets it after pull requests? Something else? :-)
Uptrenda 3y ago
Wouldn't this help solve the problem Ken Thompson wrote about in 'reflections on trusting trust?' If you can fully bootstrap a system from source code then it's harder to have things like back-doored compilers.
- raboof 3y ago
  It indeed helps, but it is not a full 'solution': you could still in theory envision elaborate backdoors in the 'environment' in which the ISO is built. If you really want to 'solve' the problem describe there, you could look into Diverse Double Compiling (https://dwheeler.com/trusting-trust/ https://dwheeler.com/trusting-trust/) or bootstrapping the entire environment (https://bootstrappable.org/ https://bootstrappable.org/) - see also the 'Aren’t there bootstrap problems with the above approach?' section of the post. Reproducing the build already goes a long way in making such attacks increasingly unlikely, though.
Crontab 3y ago
I love that there are people out there who cares about things like this.
lrvick 3y ago
Now if only they would have maintainers sign packages like almost every other linux distribution has done since the 90s, so we have any idea if the code everyone is building is the same code submitted and reviewed by known individuals. Until signing is standardized, it is hard to imagine using nix in any production use case that protects anything of value.
- __MatrixMan__ 3y ago
  My impression of nix package maintainers is that they are providing a useful interface so that software can be composed easily. Most of them are not making any kind of assertion about the contents of the package. Expecting to use their signatures for anything meaningful strikes me as a bit like expecting product support from a delivery driver. You don't need to trust it wasn't packaged maliciously, nix does reproducible builds so you can just look at the derivation and build it yourself if you don't feel like relying on the binary cache. As for whether the underlying contents are malicious, that's between you and the developer. If other distributions have have lead you to believe otherwise, then I think they have misled you. The only exception I can think of is Tails, and they don't exactly have the breadth that Nix does.
  - lrvick 3y ago
    "Expecting to use their signatures for anything meaningful strikes me as a bit like expecting product support from a delivery driver." And yet most of the packages from most major linux distributions are signed. If you are going to spend hours maintaining a package, it takes only an extra half a second to tap a yubikey to prevent someone from impersonating you. Package maintainers from say Arch and Debian go through a vetting process, multiple people sign their keys, and it is a responsibility. Yes, it is volunteer, but there are also volunteer firefighers. Some volunteer jobs are important to keep others safe, and they should be done with care. If Arch, Debian, Fedora, Ubuntu can all sign packages, then this excuse does not really hold for Nix. "You don't need to trust it wasn't packaged maliciously, nix does reproducible builds so you can just look at the derivation and build it yourself if you don't feel like relying on the binary cache." Reproducible builds and package definition signing solve totally different problems. Assume you trust a given developer has been maintaining a package non-maliciously, then you see they made a new update, and so you and other people trust it and build it. You get the same result, so you trust the binaries too. However, you still end up with malware. How? Simple. The developers github account was compromised due to a sim swap on their email account while they were on vacation, and someone pushed a fake commit as that person. Or maybe a malicious Github employee is bribed to serve manipulated git history only to the reproducible build servers but to no one else, so it goes undetected for years. Supply chain attacks like this are becoming very common, and there is a lot of motivation to do it to Linux distributions which power systems worth billions of dollars regularly. It is also so easy to close these risks. Just tap the damn yubikey or nitrokey when it blinks. It is wildly irresponsible not to mandate this, and we should question the motivations of anyone not willing to do something so basic to protect users.

548 points by CathalMullan 3y ago

deleted 3y ago

[deleted]

Sorry for being dense, but I thought one of the main reason for nixos's existence is reproducibilty. I thought they have these kinds of things solved already. I have only ~2 hours experience with Nixos, wanted to try hyprland, I thought it would be easier on Nixos since hyprland needs a bit of setup and maybe it's easier to use someone else's config on nixos, than on some other distro. Finding a config was hard too, found like 3 on some random github gists, thought there would be more... and none of them worked, at that point I gave up.

rgoulter 3y ago
Yeah, Nix is a tough tool to learn. It's probably never the right tool to pick for "I just want something that works right now" if you're unfamiliar with it. > I thought one of the main reason for nixos's existence is reproducibilty NixOS uses "reproducible" to mean "with the same Nix code, you get the same program behaviour". This is more/less what people hope Dockerfiles provide. This is the level of reproducibility you want when you say "it works on my machine" or "it worked last time I tried it". Whereas "reproducible build" aims for bit-for-bit equality for artifacts build on different machines. -- With this, there's a layer of security in that you can verify that code has been built from a particular set of sources. > Finding a config was hard too What search query were you using? Searching "nixos configuration" on https://github.com/search?q=nixos%20configuration&type=repositories https://github.com/search?q=nixos%20configuration&type=repos... Or searching for hyprland specifically, there seem to be many using that https://github.com/search?q=wayland.windowManager.hyprland&type=code https://github.com/search?q=wayland.windowManager.hyprland&t...
- amarshall 3y ago
  > NixOS uses "reproducible" to mean "with the same Nix code, you get the same program behaviour". Note that ”Nix code” also includes the hashes of all non-Nix sources. One way to think of it is that Nix has reliable build cache invalidation. > This is more/less what people hope Dockerfiles provide. Indeed, but importantly they do not provide input-reproducibility (while Nix does) because, at least, there are no hashes for remote data.
  - __MatrixMan__ 3y ago
    PSA: you can build OCI images with nix, then they'll be a pure function of their input like we've all wished was the case with Dockerfiles.
    - Smaug123 3y ago
      (And Nix derivations compose, whereas Dockerfiles entirely do not.)
- mihalycsaba 3y ago
  I don't remember, some of them needed some other tools installed(like flakes whatever it is), I looked for configs, that looked like they don't need a few more hours to learn and to setup some other tools for them to work. I just wanted to take a quick look at hyprland, I imagined I just use an existing config, I never thought it would need hours of research. Later I installed an arch vm and managed to install hyprland with some basic components in less than an hour from the first guide I found. Looks like I misunderstood, what nix was made for. I just want a system I can more or less set up with a simple config file. I saw this os, didn't have time to try it yet, but I thought this is how nix works. https://blendos.co/ https://blendos.co/ For example you just define gnome like this, the nix configs I found looked similar, they just didn't work. >gnome: > enabled: true > style: light > gtk-theme: 'adw-gtk3' > icon-theme: 'Adwaita' > titlebar: > button-placement: 'right' > double-click-action: 'toggle-maximize' > middle-click-action: 'minimize' > right-click-action: 'menu'
quietbritishjim 3y ago
There are two senses of reproducible. The sense you're thinking of is that you can easily rebuild a binary package and it will use the same dependency versions, build options, etc. There should be no chance of a compiler error that didn't happen the first time (the old "but it worked on my laptop" syndrome). The sense used here is that every build output is byte-for-byte binary identical. It doesn't depend on the machine name, the time it was compiled or anything like that (or, in a parallel build, the order in which files finish compiling). That is much harder.
- jowea 3y ago
  > The sense you're thinking of is that you can easily rebuild a binary package and it will use the same dependency versions, build options, etc. There should be no chance of a compiler error that didn't happen the first time (the old "but it worked on my laptop" syndrome). And that's just for Nixpkgs, the packages themselves that also work outside NixOS. NixOS has reproducibility of the entire system complete with configuration.
ParetoOptimal 3y ago
> Finding a config was hard too, found like 3 on some random github gists, thought there would be more.. That sounds odd, did you use github code search? Find relevant home manager options: https://mipmip.github.io/home-manager-option-search/?query=hyprland https://mipmip.github.io/home-manager-option-search/?query=h... Then search those on github: https://github.com/search?utf8=%E2%9C%93&q=lang%3Anix+hyprland+plugins&type=code https://github.com/search?utf8=%E2%9C%93&q=lang%3Anix+hyprla... Note some option searches imply more casual or advanced users.
chpatrick 3y ago
> Sorry for being dense, but I thought one of the main reason for nixos's existence is reproducibilty. I thought they have these kinds of things solved already. Nixos has the advantage that everything is built in its own sandbox with only its explicitly declared (and hashed) dependencies available, unlike in mainstream distros where it's the full system environment, so in many cases you already get the same binary every time. But this doesn't immediately lead to reproducibility because the build process might be nondeterministic for various packages.
- benreesman 3y ago
  This is a really good comment, I have no idea why it’s going grey. Upvote from me FWIW.
- WhyNotHugo 3y ago
  > unlike in mainstream distros where it's the full system environment Usually packages are built in an environment which has only a minimal base system plus the package's explicitly dependencies. They don't have random unnecessary packages installed.
- goodpoint 3y ago
  > unlike in mainstream distros Debian has been building in a clean sandbox with only required, tracked dependencies since decades. It's also building the large majority of packages reproducibly including the binary and whole installation packages (not just the sources like nixos)
  - chpatrick 3y ago
    > not just the sources like nixos Not sure what you mean by that, the Nix packages that are reproducible have reproducible binaries. In the Nixos world there isn't really a concept of a "binary/installation package" like in Debian or elsewhere. Everything can be rebuilt from source on any machine, but because everything is hashed, if the official binary caches have already built something with the same inputs, they can just give you the outputs directly. So it's more like memoization than a .deb or something that you install. Nix is a functional language that builds recipes (derivations) to build stuff, with all the inputs and outputs hashed. If the derivation you want to build has already been built by a cache you trust, the system will just fetch it instead of building locally. What the Nix reproducability project checks is that the same derivation produces the same output regardless of what machine it's built on.
Aerbil313 3y ago
Check out https://github.com/donovanglover/nix-config https://github.com/donovanglover/nix-config . Flake based config with hyprland and cool stuff. > at that point I gave up. NixOS is not for the weak or time constrained, currently. Hopefully it will be one day. Still if you push through, you reap the benefits.
- flkiwi 3y ago
  Another good option: https://github.com/Misterio77/nix-starter-configs https://github.com/Misterio77/nix-starter-configs I started with this one, the minimal version, then moved on to something more like the standard version, and now I'm moving on to something based on his much more complicated and flexible build in a different repo. I had been flailing, then this repo made it click.
  - __MatrixMan__ 3y ago
    Same here, that repo is fantastic.
    - flkiwi 3y ago
      His documentation has gotten better too! I actually just rebuilt my entire config based on his updated "standard" version. I want to use his "non-starter" config too, bc it seems remarkably powerful, but ... I need more time and brain to do that. And, just to be clear, I rebuilt my entire config in less than an hour, copying and pasting where necessary, then it took about 20 minutes (!!) to download and setup all my software. Only one error, so I consider that a significant My NixOS Journey accomplishment! Granted, because it's NixOS, I have NO IDEA AT ALL what the error was, lol.
colordrops 3y ago
Nix is reproducteable in tbe environment sense, meaning you can get the exact same setup every time, but not in the bit-for-bit sense, meaning that the compiled binaries will be identical.

Reventlov 3y ago

For those wondering : it should be remembered that the reproducibility of Nix / NixOS / Nixpkgs is only a reproducibility of the sources: if the sources change, one is warned, but it is not a question of the reproducibility of the binaries (which can change at each build). This binary reproducibility of Nix / NixOS / Nixpkgs is indeed not really tested, at least not systematically. Guix, Archlinux, Debian do the binary reproducibility better than Nix / NixOS / Nixpkgs. Sources : - https://r13y.com/ https://r13y.com/ ( Nix* ) - https://tests.reproducible-builds.org/debian/reproducible.html https://tests.reproducible-builds.org/debian/reproducible.ht... ( Debian ) - https://tests.reproducible-builds.org/archlinux/archlinux.html https://tests.reproducible-builds.org/archlinux/archlinux.ht... ( Archlinux ) - https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/package-reproducibility https://data.guix.gnu.org/repository/1/branch/master/latest-... (Guix, might be a bit slow to load, here is some cached copy https://archive.is/lTuPk https://archive.is/lTuPk )

dicytea 3y ago
> Guix, Archlinux, Debian do the binary reproducibility better than Nix / NixOS / Nixpkgs. Huh, didn't know that Arch Linux tests reproducibility. It's apparently 85.6% reproducible: https://reproducible.archlinux.org https://reproducible.archlinux.org I wonder how much work would be needed for NixOS, considering it has more than 80k packages in the official repository.
- chpatrick 3y ago
  I think that's also a bit of an unfair comparison given the number of AUR packages you usually use on Arch. With nixpkgs there isn't a distinction between official and community packages.
  - deleted 3y ago
    [deleted]
  - iopq 3y ago
    Sure there is, the NUR has a few thousand community packages that are not ready for release The nixpkgs are all official packages, it's just really easy to become a maintainer (you make a pull request adding the package you want to maintain)

onedognight 3y ago

Rebuilding the minimal ISO from source is an impressive milestone on the journey to a system that builds from source reproducibly. Guix had an orthogonal but equally impressive milestone on the same journey recently[0], bootstrapping a full compiler toolchain from a single reproducible 357 byte binary without any other binary compiler blobs. These two features may one day soon be combined to reproducibly build a full distribution from source. [0] https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-building-from-source-all-the-way-down/ https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-...

Thrir94994i 3y ago
357 bytes for bootstrap compiler binary is VERY impressive!
- msm_ 3y ago
  If I remember correctly, this tiny binary is used to (reproducibly) bootstrap the next binary, which bootstraps the next binary, until eventually GCC can be compiled (and compile other software).
  - Smaug123 3y ago
    The bootstrap chain is https://github.com/oriansj/stage0-posix-x86/blob/e86bf7d304bae5ce5ccc88454bb60cf0837e941f/mescc-tools-mini-kaem.kaem https://github.com/oriansj/stage0-posix-x86/blob/e86bf7d304b...
- rssoconnor 3y ago
  To be fair, it is 357 bytes ... plus a POSIX operating system. Still, that POSIX operating system bit is also being worked on.
  - 6581 3y ago
    Isn't that what builder-hex0 does? https://github.com/ironmeld/builder-hex0 https://github.com/ironmeld/builder-hex0

KennyFromIT 3y ago

I've lived in the Red Hat ecosystem for work recently. How does this compare to something like... Fedora Silverblue? Ansible? Fedora Silverblue + Ansible?

TheDong 3y ago
The closest equivalent to the nixos ISO builder and reproducibility related to it in the fedora ecosystem is osbuild / imagebuilder - https://www.osbuild.org/guides/introduction.html https://www.osbuild.org/guides/introduction.html Imagebuilder claims reproducibility, but as far as I know it mostly installed rpm packages as binaries, not from source, so it's not really proper reproducibility unless all the input packages are also reproducible. If the descriptions of building packages from source, building distro images, and reproducibility in the linked thread didn't make sense to you, you're probably not really the target audience anyway.
candiddevmike 3y ago
Nix is a declarative OS, where you describe what the OS should look like, instead of Ansible where you give the OS steps to follow. Silverblue and Nix are orthogonal aside from being Linux distributions--Silverblue is attempting to change how software is delivered using only containers on an immutable host. If you're interested in an Ansible alternative that uses Jsonnet and state tracking to somewhat mimic Nix, check out Etcha: https://etcha.dev https://etcha.dev
- deleted 3y ago
  [deleted]
- rgoulter 3y ago
  > Nix is a declarative OS I think precision is important. "Nix" refers to the package manager (and the language the package manager uses). Whereas it's "NixOS" that's the OS which makes use of Nix to manage the system configuration.
  - Ericson2314 3y ago

somat 3y ago

I find it funny(ironic) that the OpenBSD project is trying hard to go the other way, every single install has unique and randomized address offsets. While I understand that these two goals, reproducible builds and unique installs, are orthogonal to each other, both can be had at the same time, the duality of the situation still makes me laugh.

oever 3y ago
If the address offsets can be randomized with a provided seed, then demonstrating reproducibility is still possible. Alternatively, randomizing the offsets when starting the program is another way to keep reproducibility and even increase security; the offsets would change at every run.
WhyNotHugo 3y ago
OpenBSD does randomised linking at boot time. Packages themselves can still be reproducible. All the randomisation is done locally after the packages are downloaded and their checksums validated.

mbakke 3y ago

Very impressive milestone, congrats to those who made this possible! > [...] actually rebuilding the ISO still introduced differences. This was due to some remaining problems in the hydra cache and the way the ISO was created. Can anyone shed some light on the fix for "how the ISO was created"? I attempted making a reproducible ISO a while back but could not make the file system create extents in a deterministic fashion.

raboof 3y ago
For NixOS, it's in the 'how did we reproduce' section of the article: the last step of that process produces the iso in the ./result/iso directory. It sounds like what you're looking for is the commands that that build invoked, but I'm not sure what step you're looking for. For example, the xorriso invocations are at https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/make-iso9660-image.sh#L97 https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/make-...

ahmedfromtunis 3y ago

Stupid question as I never worked on something like this before: why isn't reproducibility the default behavior? I mean if 2 copies of a piece of software were compiled from the same source, what stops them from being identical each and every time? I know there are so many moving parts, but I still can't understand how discrepancies can manifest themselves.

bravetraveler 3y ago
I don't develop enough to give a particularly good answer, but one example I've heard of involves timestamps Imagine the program uses the current date or time as a value. When compiled at different moments, the bits change. Same applies to anything where the build environment or timing influences the output binary
kaba0 3y ago
Parallelism. There might be actions that are not order-independent, and the state of the CPU might result in slightly different binaries, but all are correct.
- edgyquant 3y ago
  Why does this matter though? Why does order of compilation result in a different binary?
  - speed_spread 3y ago
    Because order of completion of the parallel tasks is not guaranteed, if all tasks write to the same file you might get a different result each time.
  - kaba0 3y ago
    Just some random, made up example: say you want to compile an OOP PL that has interfaces and implementations of that. You discover reachable implementations through static analysis, which is multi-threaded. You might discover implementations A,B,C in any order — but they will get their methods placed in the jump table based on this order. This will trivially result in semantically equivalent, but not binary-equivalent executables. Of course there would have been better designs for this toy example, but binary reproducibility is/was usually not of the highest priority historically in most compiler infrastructures, and in some cases it might be a relatively big performance regression to fix, or simply just a too big refactor.

mgaunard 3y ago

Don't you have to fake the system time to do this? The time often ends up inside the binaries one way or another.

mbakke 3y ago
Indeed time stamps are probably the most common sources of indeterminism. So common that a de-facto standard variable to fake a timestamp has been implemented in many compilers: https://reproducible-builds.org/docs/source-date-epoch/ https://reproducible-builds.org/docs/source-date-epoch/
amelius 3y ago
Could you name an example of how (and for what reason) this might happen?
- mbakke 3y ago
  Typically part of a "version string": $ python3 Python 3.10.7 (main, Jan 1 1970, 00:00:01) [GCC 11.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> Perhaps a relic from when software had to be manually updated?
  - oever 3y ago
    On NixOS, I think the release time or commit time is used: $ python3 Python 3.10.11 (main, Apr 4 2023, 22:10:32) [GCC 12.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> That is more useful than the build time.
    - mbakke 3y ago
      How is that possible? Is nixpkgs an input to the Python derivation? Or do packagers "hard code" a value every time they modify the Python build code? Automated tooling that sets it after pull requests? Something else? :-)

Uptrenda 3y ago

Wouldn't this help solve the problem Ken Thompson wrote about in 'reflections on trusting trust?' If you can fully bootstrap a system from source code then it's harder to have things like back-doored compilers.

raboof 3y ago
It indeed helps, but it is not a full 'solution': you could still in theory envision elaborate backdoors in the 'environment' in which the ISO is built. If you really want to 'solve' the problem describe there, you could look into Diverse Double Compiling (https://dwheeler.com/trusting-trust/ https://dwheeler.com/trusting-trust/) or bootstrapping the entire environment (https://bootstrappable.org/ https://bootstrappable.org/) - see also the 'Aren’t there bootstrap problems with the above approach?' section of the post. Reproducing the build already goes a long way in making such attacks increasingly unlikely, though.

Crontab 3y ago

I love that there are people out there who cares about things like this.

lrvick 3y ago

Now if only they would have maintainers sign packages like almost every other linux distribution has done since the 90s, so we have any idea if the code everyone is building is the same code submitted and reviewed by known individuals. Until signing is standardized, it is hard to imagine using nix in any production use case that protects anything of value.

__MatrixMan__ 3y ago
My impression of nix package maintainers is that they are providing a useful interface so that software can be composed easily. Most of them are not making any kind of assertion about the contents of the package. Expecting to use their signatures for anything meaningful strikes me as a bit like expecting product support from a delivery driver. You don't need to trust it wasn't packaged maliciously, nix does reproducible builds so you can just look at the derivation and build it yourself if you don't feel like relying on the binary cache. As for whether the underlying contents are malicious, that's between you and the developer. If other distributions have have lead you to believe otherwise, then I think they have misled you. The only exception I can think of is Tails, and they don't exactly have the breadth that Nix does.
- lrvick 3y ago
  "Expecting to use their signatures for anything meaningful strikes me as a bit like expecting product support from a delivery driver." And yet most of the packages from most major linux distributions are signed. If you are going to spend hours maintaining a package, it takes only an extra half a second to tap a yubikey to prevent someone from impersonating you. Package maintainers from say Arch and Debian go through a vetting process, multiple people sign their keys, and it is a responsibility. Yes, it is volunteer, but there are also volunteer firefighers. Some volunteer jobs are important to keep others safe, and they should be done with care. If Arch, Debian, Fedora, Ubuntu can all sign packages, then this excuse does not really hold for Nix. "You don't need to trust it wasn't packaged maliciously, nix does reproducible builds so you can just look at the derivation and build it yourself if you don't feel like relying on the binary cache." Reproducible builds and package definition signing solve totally different problems. Assume you trust a given developer has been maintaining a package non-maliciously, then you see they made a new update, and so you and other people trust it and build it. You get the same result, so you trust the binaries too. However, you still end up with malware. How? Simple. The developers github account was compromised due to a sim swap on their email account while they were on vacation, and someone pushed a fake commit as that person. Or maybe a malicious Github employee is bribed to serve manipulated git history only to the reproducible build servers but to no one else, so it goes undetected for years. Supply chain attacks like this are becoming very common, and there is a lot of motivation to do it to Linux distributions which power systems worth billions of dollars regularly. It is also so easy to close these risks. Just tap the damn yubikey or nitrokey when it blinks. It is wildly irresponsible not to mandate this, and we should question the motivations of anyone not willing to do something so basic to protect users.