5 ms·

Disorderfs: FUSE-based filesystem that introduces non-determinism into metadata

127 points by mmastrac 6y ago

izietto 6y ago
On Gitlab isn't possible even to read the Readme without javascript enabled. Too bad!
- kenniskrag 6y ago
  https://salsa.debian.org/reproducible-builds/disorderfs/-/raw/master/README https://salsa.debian.org/reproducible-builds/disorderfs/-/ra...
EdSchouten 6y ago
Buildbarn, a build cluster implementation for Bazel that I maintain, can also run build actions (compilation steps, unit tests) in a FUSE file system. Though the primary motivator for this is that it reduces the time to construct a build action's file system to nearly instant, it has the advantage that I can also do things similar to disorderfs. Shuffling directory listings is actually something that I also added. Pretty useful! https://github.com/buildbarn/bb-remote-execution/blob/eb11502902afa733356d7f2027bd41ed3872766e/pkg/proto/configuration/fuse/fuse.proto#L59-L70 https://github.com/buildbarn/bb-remote-execution/blob/eb1150...
- infogulch 6y ago
  If you are confident that certain build steps are deterministic, can you enable Dockerfile-like caching for intermediate steps? The way docker does it is take a hash of the "input" filesystem and the command, and see if there's an associated "result" filesystem, and if there is then just jump to evaluating the next command with the previous result as input.
  - ithkuil 6y ago
    That's precisely what bazel does (and remote build execution systems like buildbarn allow this cache to be effectively shared among users)
    - infogulch 6y ago
      Ah, cool! IIRC Bazel predates docker, so perhaps it's more correct to say that dockerfiles use a Bazel-like command caching strategy.
      - ithkuil 6y ago
        Yep, moreover bazel has a fine grained dependency graph and thus can invalidate only a minimal part of the action graph when some inputs change. Docker instead has to re-build all subsequent layers when the input of even just one layer changes. The two tools sit on a different point in the spectrum of simplicity of use though. Maintain build files for bazel (and dealing with other constraints of hermetic execution) is hard time-consuming and it's hard to convince many teams it's worth the effort. Docker apparently struck a sweet spot in that it's easy to explain how to craft linear build steps and it does a half-decent job in actually caching stuff. Sometimes it doesn't work well depending on your workflow but people then have the incentive to read about how to improve the "cacheability" of their dockerfiles (multistage, reorder, dockerignore, ...). As many things in our craft, the human aspect trumps over technical brilliance.
danbst 6y ago
Sorry for plugging in. NixOS has a plan to use DisorderFS in making 100% reproducible build. https://r13y.com/ https://r13y.com/ As I understand it, currently several packages are not reproducible (like, python, pytest, gcc), so it is not priority, but when those large packages will be done, r13y will start using DisorderFS to uncover remaining reproducibility bugs. This is too idealistic, but gives lots of pleasure about package space.
- amelius 6y ago
  Instead of introducing non-determinism, shouldn't it instead try to enforce determinism in any possible way? (E.g. running all processes under ptrace or by using virtualization and thereby making the OS behave in a deterministic way during a build).
  - rblatz 6y ago
    I think the idea is you can build from source on any file system or drive and get the same results. By adding artificial purposeful nondeterminism you can fix your builds to account for unintentional natural nondeterminism.
  - ithkuil 6y ago
    The goal is to ensure that builds can be deterministic despite non-determinism. Once such criterion is enforced, then everybody can reproduce the build with that set of source files and build instructions, without requiring a special environment that forces a specific order of events.
    - lambda_obrien 6y ago
      > The goal is to ensure that builds can be deterministic despite non-determinism ...by using non-determinism? Very mind bending for me, I'm not sure I understand, but I'm glad smart people are figuring this stuff out.
      - agwa 6y ago
eeZah7Ux 6y ago
The work done in Debian around reproducible builds is really impressive.
myself248 6y ago
From the headline, I was picturing EXIF tags, which would be pretty amusing. Store your photos at this mountpoint here, read 'em from that mountpoint over there, and their locations and camera details get anonymized. But the real data still lives on disk. Alas that's not what this is about, but now I wonder how hard it would be to make the thing I had in my head.
- rkeene2 6y ago
  Not too hard. You should build it !
- mceachen 6y ago
  A FUSE mount, or even a webdav virtual directory hierarchy, wouldn't be that tricky. Stripping metadata is trivial, thanks to ExifTool.
nixbuild 6y ago
nixbuild.net ("Nix Builds as a Service") can run any Nix build with fs randomization similar to disorderfs: https://blog.nixbuild.net/posts/2021-01-13-finding-non-determinism-with-nixbuild-net.html https://blog.nixbuild.net/posts/2021-01-13-finding-non-deter... It isn't enabled by default but can be turned on with a setting: https://docs.nixbuild.net/settings/#inject-fs-randomness https://docs.nixbuild.net/settings/#inject-fs-randomness

127 points by mmastrac 6y ago

izietto 6y ago

On Gitlab isn't possible even to read the Readme without javascript enabled. Too bad!

kenniskrag 6y ago
https://salsa.debian.org/reproducible-builds/disorderfs/-/raw/master/README https://salsa.debian.org/reproducible-builds/disorderfs/-/ra...

EdSchouten 6y ago

Buildbarn, a build cluster implementation for Bazel that I maintain, can also run build actions (compilation steps, unit tests) in a FUSE file system. Though the primary motivator for this is that it reduces the time to construct a build action's file system to nearly instant, it has the advantage that I can also do things similar to disorderfs. Shuffling directory listings is actually something that I also added. Pretty useful! https://github.com/buildbarn/bb-remote-execution/blob/eb11502902afa733356d7f2027bd41ed3872766e/pkg/proto/configuration/fuse/fuse.proto#L59-L70 https://github.com/buildbarn/bb-remote-execution/blob/eb1150...

infogulch 6y ago
If you are confident that certain build steps are deterministic, can you enable Dockerfile-like caching for intermediate steps? The way docker does it is take a hash of the "input" filesystem and the command, and see if there's an associated "result" filesystem, and if there is then just jump to evaluating the next command with the previous result as input.
- ithkuil 6y ago
  That's precisely what bazel does (and remote build execution systems like buildbarn allow this cache to be effectively shared among users)
  - infogulch 6y ago
    Ah, cool! IIRC Bazel predates docker, so perhaps it's more correct to say that dockerfiles use a Bazel-like command caching strategy.
    - ithkuil 6y ago
      Yep, moreover bazel has a fine grained dependency graph and thus can invalidate only a minimal part of the action graph when some inputs change. Docker instead has to re-build all subsequent layers when the input of even just one layer changes. The two tools sit on a different point in the spectrum of simplicity of use though. Maintain build files for bazel (and dealing with other constraints of hermetic execution) is hard time-consuming and it's hard to convince many teams it's worth the effort. Docker apparently struck a sweet spot in that it's easy to explain how to craft linear build steps and it does a half-decent job in actually caching stuff. Sometimes it doesn't work well depending on your workflow but people then have the incentive to read about how to improve the "cacheability" of their dockerfiles (multistage, reorder, dockerignore, ...). As many things in our craft, the human aspect trumps over technical brilliance.

danbst 6y ago

Sorry for plugging in. NixOS has a plan to use DisorderFS in making 100% reproducible build. https://r13y.com/ https://r13y.com/ As I understand it, currently several packages are not reproducible (like, python, pytest, gcc), so it is not priority, but when those large packages will be done, r13y will start using DisorderFS to uncover remaining reproducibility bugs. This is too idealistic, but gives lots of pleasure about package space.

amelius 6y ago
Instead of introducing non-determinism, shouldn't it instead try to enforce determinism in any possible way? (E.g. running all processes under ptrace or by using virtualization and thereby making the OS behave in a deterministic way during a build).
- rblatz 6y ago
  I think the idea is you can build from source on any file system or drive and get the same results. By adding artificial purposeful nondeterminism you can fix your builds to account for unintentional natural nondeterminism.
- ithkuil 6y ago
  The goal is to ensure that builds can be deterministic despite non-determinism. Once such criterion is enforced, then everybody can reproduce the build with that set of source files and build instructions, without requiring a special environment that forces a specific order of events.
  - lambda_obrien 6y ago
    > The goal is to ensure that builds can be deterministic despite non-determinism ...by using non-determinism? Very mind bending for me, I'm not sure I understand, but I'm glad smart people are figuring this stuff out.
    - agwa 6y ago

eeZah7Ux 6y ago

The work done in Debian around reproducible builds is really impressive.

myself248 6y ago

From the headline, I was picturing EXIF tags, which would be pretty amusing. Store your photos at this mountpoint here, read 'em from that mountpoint over there, and their locations and camera details get anonymized. But the real data still lives on disk. Alas that's not what this is about, but now I wonder how hard it would be to make the thing I had in my head.

rkeene2 6y ago
Not too hard. You should build it !
mceachen 6y ago
A FUSE mount, or even a webdav virtual directory hierarchy, wouldn't be that tricky. Stripping metadata is trivial, thanks to ExifTool.

nixbuild 6y ago

nixbuild.net ("Nix Builds as a Service") can run any Nix build with fs randomization similar to disorderfs: https://blog.nixbuild.net/posts/2021-01-13-finding-non-determinism-with-nixbuild-net.html https://blog.nixbuild.net/posts/2021-01-13-finding-non-deter... It isn't enabled by default but can be turned on with a setting: https://docs.nixbuild.net/settings/#inject-fs-randomness https://docs.nixbuild.net/settings/#inject-fs-randomness