6 ms·

Sorry for plugging in. NixOS has a plan to use DisorderFS in making 100% reproducible build. https://r13y.com/ https://r13y.com/ As I understand it, currently

by danbst 6y ago

Sorry for plugging in. NixOS has a plan to use DisorderFS in making 100% reproducible build. https://r13y.com/ https://r13y.com/ As I understand it, currently several packages are not reproducible (like, python, pytest, gcc), so it is not priority, but when those large packages will be done, r13y will start using DisorderFS to uncover remaining reproducibility bugs. This is too idealistic, but gives lots of pleasure about package space.

amelius 6y ago
Instead of introducing non-determinism, shouldn't it instead try to enforce determinism in any possible way? (E.g. running all processes under ptrace or by using virtualization and thereby making the OS behave in a deterministic way during a build).
- rblatz 6y ago
  I think the idea is you can build from source on any file system or drive and get the same results. By adding artificial purposeful nondeterminism you can fix your builds to account for unintentional natural nondeterminism.
- ithkuil 6y ago
  The goal is to ensure that builds can be deterministic despite non-determinism. Once such criterion is enforced, then everybody can reproduce the build with that set of source files and build instructions, without requiring a special environment that forces a specific order of events.
  - lambda_obrien 6y ago
    > The goal is to ensure that builds can be deterministic despite non-determinism ...by using non-determinism? Very mind bending for me, I'm not sure I understand, but I'm glad smart people are figuring this stuff out.
    - agwa 6y ago
      You use disorderfs as part of a CI process. The CI builds the package once without disorderfs, and once with disorderfs. If they produce the same output, the package is reproducible (at least with respect to filesystem order). Otherwise, something in the build process is depending on filesystem order and should be fixed to sort directory entries before using them. You wouldn't use disorderfs when building a package normally. (At least this is how Debian uses disorderfs. I wrote the first version of disorderfs 6 years ago in a hacking session at DebConf15 in Heidelberg. I never expected to see it on the front page of HN!)
      - lambda_obrien 6y ago
        Thanks! That makes more sense now.
      - amelius 6y ago
        I think you need to build many times to be sure. Therefore (the original question), instead of using "disorderfs", why not write and use an "orderedfs" for every build?
        cyphar 6y ago
        Because doing it that way will make building (and verifying) a deterministic build more difficult for users, while forcing the builds to be deterministic in the face of randomised non-determinism means that anyone can build the project and get the same output without needing any complicated build configuration. That end goal (all builds are deterministic even if you don't have some magical reproducible build machine) is the holy grail of reproducible builds. And since this is run as part of a CI process, you will get lots of builds over time and will root out all sorts of issues caused by non-determinism.
        agwa 6y ago
        The original behavior of disorderfs was to randomly shuffle directory entries, but we quickly realized that this meant that sometimes the shuffle wouldn't do anything, so I changed the default behavior to simply reverse the directory entries instead. Therefore, you only have to build twice. (Ironically, disorderfs' "non-determinism" is actually deterministic.) As to your original question, there are so many sources of nondeterminism that trying to emulate them all away would make builds more complicated, less performant (FUSE adds overhead), and less safe (since there would be more components that could potentially be backdoored).
- Spivak 6y ago
  It’s going to be part of the testing suite. The packages should be able to build exactly the same even in a hostile environment. You want the environment “outside” your build system to be as chaotic as possible so you know there aren’t any accidental dependencies you don’t catch.
- gopalv 6y ago
  > shouldn't it instead try to enforce determinism in any possible way That would only fix the build machine's problem, it wouldn't fix anyone else's builds. A repeatable build without determinism is a fix for all people everywhere.
  - amelius 6y ago
    > That would only fix the build machine's problem, it wouldn't fix anyone else's builds. It would fix everyone else's builds if they added determinism in their builds (as opposed to adding non-determinism in the test-procedure). Besides, a test-procedure never gives a guarantee because a bug depending on non-determinism can be subtle.
    - _underfl0w_ 6y ago
      > a bug depending on non-determinism can be subtle. That's the point - to uncover bugs dependent on nondeterminism by using a filesystem that introduces it. This is for fuzz testing at the filesystem level, not literally reproducing the builds correctly multiple times. From the linked README: "This is useful for detecting non-determinism in the build process."
- ampdepolymerase 6y ago
  It's a form of fuzzing. A bug free build system would not be affected by non-determinism.
- deleted 6y ago
  [deleted]
Foxboron 6y ago
Strictly speaking, this is nothing new. Debian has been doing it for 5 years. https://reproducible-builds.org/citests/ https://reproducible-builds.org/citests/ All distros there are essentially reproducible fuzzing CI systems that introduces determinism through disorderfs, lang changes and so on. These changes are fine to fix but nothing you'd normally get when reproducing packages for a distribution. Personally the important part is if the patches are upstreamed or not. This isn't something that is a priority among distribution. Results from a fuzzing in Arch: https://tests.reproducible-builds.org/archlinux/archlinux.html https://tests.reproducible-builds.org/archlinux/archlinux.ht... Results from just chroot recreation: https://reproducible.archlinux.org https://reproducible.archlinux.org

by danbst 6y ago

amelius 6y ago

Instead of introducing non-determinism, shouldn't it instead try to enforce determinism in any possible way? (E.g. running all processes under ptrace or by using virtualization and thereby making the OS behave in a deterministic way during a build).

rblatz 6y ago
I think the idea is you can build from source on any file system or drive and get the same results. By adding artificial purposeful nondeterminism you can fix your builds to account for unintentional natural nondeterminism.
ithkuil 6y ago
The goal is to ensure that builds can be deterministic despite non-determinism. Once such criterion is enforced, then everybody can reproduce the build with that set of source files and build instructions, without requiring a special environment that forces a specific order of events.
- lambda_obrien 6y ago
  > The goal is to ensure that builds can be deterministic despite non-determinism ...by using non-determinism? Very mind bending for me, I'm not sure I understand, but I'm glad smart people are figuring this stuff out.
  - agwa 6y ago
    You use disorderfs as part of a CI process. The CI builds the package once without disorderfs, and once with disorderfs. If they produce the same output, the package is reproducible (at least with respect to filesystem order). Otherwise, something in the build process is depending on filesystem order and should be fixed to sort directory entries before using them. You wouldn't use disorderfs when building a package normally. (At least this is how Debian uses disorderfs. I wrote the first version of disorderfs 6 years ago in a hacking session at DebConf15 in Heidelberg. I never expected to see it on the front page of HN!)
    - lambda_obrien 6y ago
      Thanks! That makes more sense now.
    - amelius 6y ago
      I think you need to build many times to be sure. Therefore (the original question), instead of using "disorderfs", why not write and use an "orderedfs" for every build?
      - cyphar 6y ago
        Because doing it that way will make building (and verifying) a deterministic build more difficult for users, while forcing the builds to be deterministic in the face of randomised non-determinism means that anyone can build the project and get the same output without needing any complicated build configuration. That end goal (all builds are deterministic even if you don't have some magical reproducible build machine) is the holy grail of reproducible builds. And since this is run as part of a CI process, you will get lots of builds over time and will root out all sorts of issues caused by non-determinism.
      - agwa 6y ago
        The original behavior of disorderfs was to randomly shuffle directory entries, but we quickly realized that this meant that sometimes the shuffle wouldn't do anything, so I changed the default behavior to simply reverse the directory entries instead. Therefore, you only have to build twice. (Ironically, disorderfs' "non-determinism" is actually deterministic.) As to your original question, there are so many sources of nondeterminism that trying to emulate them all away would make builds more complicated, less performant (FUSE adds overhead), and less safe (since there would be more components that could potentially be backdoored).
Spivak 6y ago
It’s going to be part of the testing suite. The packages should be able to build exactly the same even in a hostile environment. You want the environment “outside” your build system to be as chaotic as possible so you know there aren’t any accidental dependencies you don’t catch.
gopalv 6y ago
> shouldn't it instead try to enforce determinism in any possible way That would only fix the build machine's problem, it wouldn't fix anyone else's builds. A repeatable build without determinism is a fix for all people everywhere.
- amelius 6y ago
  > That would only fix the build machine's problem, it wouldn't fix anyone else's builds. It would fix everyone else's builds if they added determinism in their builds (as opposed to adding non-determinism in the test-procedure). Besides, a test-procedure never gives a guarantee because a bug depending on non-determinism can be subtle.
  - _underfl0w_ 6y ago
    > a bug depending on non-determinism can be subtle. That's the point - to uncover bugs dependent on nondeterminism by using a filesystem that introduces it. This is for fuzz testing at the filesystem level, not literally reproducing the builds correctly multiple times. From the linked README: "This is useful for detecting non-determinism in the build process."
ampdepolymerase 6y ago
It's a form of fuzzing. A bug free build system would not be affected by non-determinism.
deleted 6y ago
[deleted]

Foxboron 6y ago

Strictly speaking, this is nothing new. Debian has been doing it for 5 years. https://reproducible-builds.org/citests/ https://reproducible-builds.org/citests/ All distros there are essentially reproducible fuzzing CI systems that introduces determinism through disorderfs, lang changes and so on. These changes are fine to fix but nothing you'd normally get when reproducing packages for a distribution. Personally the important part is if the patches are upstreamed or not. This isn't something that is a priority among distribution. Results from a fuzzing in Arch: https://tests.reproducible-builds.org/archlinux/archlinux.html https://tests.reproducible-builds.org/archlinux/archlinux.ht... Results from just chroot recreation: https://reproducible.archlinux.org https://reproducible.archlinux.org