4 ms·

> Our most important finding is that the reproducibility rate in nixpkgs has increased steadily from 69% in 2017 to about 91% in April 2023. The high reproducib

> Our most important finding is that the reproducibility rate in nixpkgs has increased steadily from 69% in 2017 to about 91% in April 2023. The high reproducibility rate in our most recent revision is quite impressive, given both the size of the package set and the absence of systematic monitoring in nixpkgs. That's one way to read the statistic. Another way you could read the graph is that they still have about the same number (~5k) of non-reproducible builds, which has been pretty constant over the time period. Adding a bunch of easily reproducible additional builds maybe doesn't make me believe it's solving the original issues. > We knew that it was possible to achieve very good reproducibility rate in smaller package sets like Debian, but this shows that achieving very high bitwise reproducibility is possible at scale, something that was believed impossible by practitioners. Maybe I miss some nuance here, but why is Debian written off as being so much smaller scale? The top end of the graph here suggests a bit over 70k packages, Debian apparently also currently has 74k packages available (https://www.debian.org/doc/manuals/debian-reference/ch02.en.html https://www.debian.org/doc/manuals/debian-reference/ch02.en....); I guess there's maybe a bit of time lag here but I'm not sure that is enough to claim Debian is somehow not "at scale".

dartos 2y ago
Are they mostly the same 5k packages as 2017? That seems to be the crux of it.
vlovich123 2y ago
According to https://tests.reproducible-builds.org/debian/reproducible.html https://tests.reproducible-builds.org/debian/reproducible.ht... (which is what the article links to btw) there are ~37k packages tracked for reproducible builds which is ~2.7x smaller than Nix's 100k packages.
gf000 2y ago
This is not really a Nix-issue to begin with. It's a bit like asking what percentage of Nix-packaged programs have Hungarian translation -- if Nix packages some more stuff the rate might decrease, but it's not Nix's task to add that to the programs that lack it. Nix does everything in its power to provide a sandboxed environment in which builds can happen. Given the way hardware works, there are still sources of non-determinism that are impossible to prevent, most importantly timing. Most programs depend on it, even compilers, and extra care should be taken by them to change that. The only way to prevent it would be to go full-on CPU and OS emulation, but that would be prohibitively expensive.
- genewitch 2y ago
  > The only way to prevent it would be to go full-on CPU and OS emulation, but that would be prohibitively expensive. How so? For ref I used a Gentoo distcc chroot inside a devuan VM to bootstrap gentoo on a 2009 netbook. It worked fine. I did this around Halloween.
  - gf000 2y ago
    A compiler invoked twice on the same source file is not mandated to produce the same binary, but it should produce a binary with the same functionality. There are infinite number of binaries that do the same thing (e.g. just padding random zeros in certain places wouldn't cause a functional problem). Nix is very good at doing functionally reproducible builds, that's its whole thing. But there are build steps which are simply not deterministic, and they might produce correct, but not always the same outputs.
  - orbital-decay 2y ago
    OS scheduling is non-deterministic, and there are quite a few things that are sensitive to the order of operations (simplest example: floating point addition). Is you want to guarantee determinism, not just provide it on a best effort basis for things that are willing to cooperate, the only way do that is to put everything into a fully deterministic emulator, which is terribly slow.
    - genewitch 2y ago
      is SCHED_FIFO or SCHED_RR or the newer incarnations viable? Is it possible to make QEMU deterministic? a quick glance at the realtime schedulers state source code needs to take that into consideration. I know it seems like i don't know what i am talking about, often, on here. The reason is i don't live on HN so i type a comment and rarely remember the words i want to use before the edit/delete window closes. The idea that i can't control when things occur during a compilation seems suspect. Is there a certain "code size" or other bellwether where this non-determinism starts to crop up? I ask because i get the feeling if i start compiling trivial stuff it will all be bit-perfect and if i say so i'll get lambasted for "well obviously trivial stuff can be reproducible," so i am heading that off at the pass, first.

dartos 2y ago

Are they mostly the same 5k packages as 2017? That seems to be the crux of it.

vlovich123 2y ago

According to https://tests.reproducible-builds.org/debian/reproducible.html https://tests.reproducible-builds.org/debian/reproducible.ht... (which is what the article links to btw) there are ~37k packages tracked for reproducible builds which is ~2.7x smaller than Nix's 100k packages.

gf000 2y ago

This is not really a Nix-issue to begin with. It's a bit like asking what percentage of Nix-packaged programs have Hungarian translation -- if Nix packages some more stuff the rate might decrease, but it's not Nix's task to add that to the programs that lack it. Nix does everything in its power to provide a sandboxed environment in which builds can happen. Given the way hardware works, there are still sources of non-determinism that are impossible to prevent, most importantly timing. Most programs depend on it, even compilers, and extra care should be taken by them to change that. The only way to prevent it would be to go full-on CPU and OS emulation, but that would be prohibitively expensive.

genewitch 2y ago
> The only way to prevent it would be to go full-on CPU and OS emulation, but that would be prohibitively expensive. How so? For ref I used a Gentoo distcc chroot inside a devuan VM to bootstrap gentoo on a 2009 netbook. It worked fine. I did this around Halloween.
- gf000 2y ago
  A compiler invoked twice on the same source file is not mandated to produce the same binary, but it should produce a binary with the same functionality. There are infinite number of binaries that do the same thing (e.g. just padding random zeros in certain places wouldn't cause a functional problem). Nix is very good at doing functionally reproducible builds, that's its whole thing. But there are build steps which are simply not deterministic, and they might produce correct, but not always the same outputs.
- orbital-decay 2y ago
  OS scheduling is non-deterministic, and there are quite a few things that are sensitive to the order of operations (simplest example: floating point addition). Is you want to guarantee determinism, not just provide it on a best effort basis for things that are willing to cooperate, the only way do that is to put everything into a fully deterministic emulator, which is terribly slow.
  - genewitch 2y ago
    is SCHED_FIFO or SCHED_RR or the newer incarnations viable? Is it possible to make QEMU deterministic? a quick glance at the realtime schedulers state source code needs to take that into consideration. I know it seems like i don't know what i am talking about, often, on here. The reason is i don't live on HN so i type a comment and rarely remember the words i want to use before the edit/delete window closes. The idea that i can't control when things occur during a compilation seems suspect. Is there a certain "code size" or other bellwether where this non-determinism starts to crop up? I ask because i get the feeling if i start compiling trivial stuff it will all be bit-perfect and if i say so i'll get lambasted for "well obviously trivial stuff can be reproducible," so i am heading that off at the pass, first.