4 ms·

Great coverage, however it failed to mention code review and artifact signing as well as full source bootstrapping which are fundamental defenses most distros s

by lrvick 2y ago

Great coverage, however it failed to mention code review and artifact signing as well as full source bootstrapping which are fundamental defenses most distros skip. In our distro, Stagex, our threat model assumes at least one maintainer, sysadmin, or computer is compromised at all times. This has resulted in some specific design choices and practices: - 100% deterministic, hermetic, reproducible - full source bootstrapped from 180 bytes of human-auditable machine code - all commits signed by authors - all reviews signed by reviewers - all released artifacts are multi-party reproduced and signed - fully OCI (container) native all the way down "FROM scratch" - All packages easily hash-locked to give downstream software easy determinism as well This all goes well beyond the tactics used in Nix and Guix. As far as we know, Stagex is the only distro designed to strictly distrust maintainers. https://stagex.tools https://stagex.tools

AstralStorm 2y ago
Good step. It doesn't distrust the developers of the software though, so does not fix the biggest hole. Multiparty reproduction does not fix it either, that only distrusts the build system. The bigger the project, the higher the chance something slips through, if even an exploitable bug. Maybe it's the developer themselves being compromised, or their maintainer. Reviews are done on what, you have someone reviewing clang code? Binutils?
- TacticalCoder 2y ago
  > Reviews are done on what, you have someone reviewing clang code? Binutils? There aren't random developers pushing commits to these codebases: these are used by virtually every Linux distro out there (OK, maybe not the Kubernetes one that ships only 12 binaries, forgot its name). It seems obvious to me that GP is talking about protection against rogue distro maintainers, not fundamental packages being backdoored. You're basically saying: "GP's work is pointless because Linus could insert a backdoor in the Linux kernel". In addition to that determinism and 100% reproducibility brings another gigantic benefit: should a backdoor ever be found in clang or one of the binutils tool, it's going to be 100% reproducible. And that is a big thing: being able to reproduce a backdoor is a godsend for security.
  - lrvick 2y ago
    > OK, maybe not the Kubernetes one that ships only 12 binaries, forgot its name You are likely thinking of Talos Linux, which incidentally also builds itself with stagex.
- lrvick 2y ago
  As the other (dead, but correct) commenter pointed out, job one is proving the released binary artifacts even match source code, as that is the spot that is most opaque to the public where vulns can most easily be injected (and have been in the past over and over and over). Only with this problem solved, can we prove the code humans ideally start spending a lot more time reviewing (working on it) is actually the code that is shipped in compiled artifacts.
  - charcircuit 2y ago
    >can most easily be injected (and have been in the past over and over and over). In practice this is much more rare then a user downloading and running malware or visiting a site that exploits their browser. Compare the number of 0days chrome has had over the years versus the number of times bad actors have hacked Google and replaced download links with links to malware.
    - lrvick 2y ago
      Nothing can stop users from being tricked, but normalizing the expectation of signing is our best defense. For instance, we trained users to start to expect the green lock, and started normalizing passkeys and fido2 which prove you are on the correct domain, taking phishing off the table. Non-web software distribution, particularly for developers, has failed to mature significantly here. Most developers today use brew, nix, alpine, dockerhub, etc. None are signed in a way that allows end users to automatically prove they got artifacts that were faithfully and deterministically built from the expected source code. Could be malware, could be anything. The typical blind trust contract from developers to CDNs that host final compiled artifacts baffles me. Of course you will get malware this way. Stagex by contrast uses OCI standard signing, meaning you can optionally set a containers/policy.json file in docker or whatever container runtime you use that will cause it to refuse to run any stagex images without reproduction signatures by two or more maintainers. If you choose to, you can automatically rule out any single developer or system in the stagex chain from injecting malware into your projects.
      - charcircuit 2y ago
- pabs3 2y ago
  The code review problem is something solvable by something like CREV, where the developer community at large publishes the reviews they have done, and eventually there is good coverage of most things. https://github.com/crev-dev/ https://github.com/crev-dev/
no-dr-onboard 2y ago
100% reproducible? That's amazing. I'll be honest, I don't really believe you (which I suppose is the point, right?). Do you all document how you got around system level sources of non-determinism? Filesystems, metadata, timestamps, tempfiles, etc? This would be a great thing to document for people aiming for the same thing. What are you all using to verify commits? Are you guys verifying signatures against a public PKI? Super interested as I manage the reproducibility program for a large software company.
- pabs3 2y ago
  Read through these websites and LWN articles: https://reproducible-builds.org/ https://reproducible-builds.org/ https://bootstrappable.org/ https://bootstrappable.org/ https://bootstrapping.miraheze.org/ https://bootstrapping.miraheze.org/ https://lwn.net/Articles/983340/ https://lwn.net/Articles/983340/ https://lwn.net/Articles/985739/ https://lwn.net/Articles/985739/
- lrvick 2y ago
  Indeed you do not have to believe me. > git clone https://codeberg.org/stagex/stagex https://codeberg.org/stagex/stagex > cd stagex > make Several hours later your "out" directory will contain locally built OCI images for every package in the tree, and the index.json for each should contain the exact same digests we commit in the "digests" folder, and the same ones multiple maintainers sign in the OCI standard "signatures" folder. We build with only a light make wrapper around docker today, though it assumes you have it configured to use the containerd image store backend, which allows for getting deterministic local digests without uploading to a registry. No reason you cannot build with podman or kaniko etc with some tweaks (which we hope to support officially) > Do you all document how you got around system level sources of non-determinism? Filesystems, metadata, timestamps, tempfiles, etc? This would be a great thing to document for people aiming for the same thing. We try to keep our package definitions to "FROM scratch" in "linux from scratch" style with no magic to be self documenting to be easy to audit or reference. By all means crib any of our tactics. We use no global env, so each package has only the determinism tweaks needed (if any). We heavily referenced Alpine, Arch, Mirage, Guix, Nix, and Debian to arrive at our current patterns. > What are you all using to verify commits? Are you guys verifying signatures against a public PKI? We all sign commits, reviews, and releases with well published PGP keys maintained in smartcards, with expected public keys in the MAINTAINERS file. Most of us have keyoxide profiles as well making it easy to prove all our online presences agree with the expected fingerprints for us. > Super interested as I manage the reproducibility program for a large software company. By all means drop in our matrix room, #stagex:matrix.org . Not many people working on these problems. The more we can all collaborate to unblock each other the better!
floxy 2y ago
>full source bootstrapped from 180 bytes of human-auditable machine code What does this mean? You have a C-like compiler in 180 bytes of assembler that can compile a C compiler that can then compile GCC?
- mananaysiempre 2y ago
  That’s normally what this means, yes, with a few more intermediate steps. There’s only one bootstrap chain like this that I know of[1,2,3], maintained by Jeremiah Orians and the Guix project; judging from the reference to 180 bytes, that’s what the distro GP describes is using as well. > This is a set of manually created hex programs in a Cthulhu Path to madness fashion. Which only have the goal of creating a bootstrapping path to a C compiler capable of compiling GCC, with only the explicit requirement of a single 1 KByte binary or less. [1] https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-building-from-source-all-the-way-down/ https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-... [2] https://savannah.nongnu.org/projects/stage0/ https://savannah.nongnu.org/projects/stage0/ [3] https://github.com/oriansj/bootstrap-seeds https://github.com/oriansj/bootstrap-seeds
  - floxy 2y ago
    That's pretty awesome
    - lrvick 2y ago
      Yep, Guix and stagex are the only two distros that full source bootstrap to my knowleldge. We use an abbreviated and explicit stage0 chain here for easy auditing: https://codeberg.org/stagex/stagex/src/branch/main/packages/bootstrap/stage0/Containerfile https://codeberg.org/stagex/stagex/src/branch/main/packages/...
      - pabs3 2y ago
        IIRC the FreeDesktop flatpak runtimes are also built from the Bootstrappable Builds folks full source bootstrap.
deleted 2y ago
[deleted]

by lrvick 2y ago

AstralStorm 2y ago

Good step. It doesn't distrust the developers of the software though, so does not fix the biggest hole. Multiparty reproduction does not fix it either, that only distrusts the build system. The bigger the project, the higher the chance something slips through, if even an exploitable bug. Maybe it's the developer themselves being compromised, or their maintainer. Reviews are done on what, you have someone reviewing clang code? Binutils?

TacticalCoder 2y ago
> Reviews are done on what, you have someone reviewing clang code? Binutils? There aren't random developers pushing commits to these codebases: these are used by virtually every Linux distro out there (OK, maybe not the Kubernetes one that ships only 12 binaries, forgot its name). It seems obvious to me that GP is talking about protection against rogue distro maintainers, not fundamental packages being backdoored. You're basically saying: "GP's work is pointless because Linus could insert a backdoor in the Linux kernel". In addition to that determinism and 100% reproducibility brings another gigantic benefit: should a backdoor ever be found in clang or one of the binutils tool, it's going to be 100% reproducible. And that is a big thing: being able to reproduce a backdoor is a godsend for security.
- lrvick 2y ago
  > OK, maybe not the Kubernetes one that ships only 12 binaries, forgot its name You are likely thinking of Talos Linux, which incidentally also builds itself with stagex.
lrvick 2y ago
As the other (dead, but correct) commenter pointed out, job one is proving the released binary artifacts even match source code, as that is the spot that is most opaque to the public where vulns can most easily be injected (and have been in the past over and over and over). Only with this problem solved, can we prove the code humans ideally start spending a lot more time reviewing (working on it) is actually the code that is shipped in compiled artifacts.
- charcircuit 2y ago
  >can most easily be injected (and have been in the past over and over and over). In practice this is much more rare then a user downloading and running malware or visiting a site that exploits their browser. Compare the number of 0days chrome has had over the years versus the number of times bad actors have hacked Google and replaced download links with links to malware.
  - lrvick 2y ago
    Nothing can stop users from being tricked, but normalizing the expectation of signing is our best defense. For instance, we trained users to start to expect the green lock, and started normalizing passkeys and fido2 which prove you are on the correct domain, taking phishing off the table. Non-web software distribution, particularly for developers, has failed to mature significantly here. Most developers today use brew, nix, alpine, dockerhub, etc. None are signed in a way that allows end users to automatically prove they got artifacts that were faithfully and deterministically built from the expected source code. Could be malware, could be anything. The typical blind trust contract from developers to CDNs that host final compiled artifacts baffles me. Of course you will get malware this way. Stagex by contrast uses OCI standard signing, meaning you can optionally set a containers/policy.json file in docker or whatever container runtime you use that will cause it to refuse to run any stagex images without reproduction signatures by two or more maintainers. If you choose to, you can automatically rule out any single developer or system in the stagex chain from injecting malware into your projects.
    - charcircuit 2y ago
pabs3 2y ago
The code review problem is something solvable by something like CREV, where the developer community at large publishes the reviews they have done, and eventually there is good coverage of most things. https://github.com/crev-dev/ https://github.com/crev-dev/

no-dr-onboard 2y ago

100% reproducible? That's amazing. I'll be honest, I don't really believe you (which I suppose is the point, right?). Do you all document how you got around system level sources of non-determinism? Filesystems, metadata, timestamps, tempfiles, etc? This would be a great thing to document for people aiming for the same thing. What are you all using to verify commits? Are you guys verifying signatures against a public PKI? Super interested as I manage the reproducibility program for a large software company.

pabs3 2y ago
Read through these websites and LWN articles: https://reproducible-builds.org/ https://reproducible-builds.org/ https://bootstrappable.org/ https://bootstrappable.org/ https://bootstrapping.miraheze.org/ https://bootstrapping.miraheze.org/ https://lwn.net/Articles/983340/ https://lwn.net/Articles/983340/ https://lwn.net/Articles/985739/ https://lwn.net/Articles/985739/
lrvick 2y ago
Indeed you do not have to believe me. > git clone https://codeberg.org/stagex/stagex https://codeberg.org/stagex/stagex > cd stagex > make Several hours later your "out" directory will contain locally built OCI images for every package in the tree, and the index.json for each should contain the exact same digests we commit in the "digests" folder, and the same ones multiple maintainers sign in the OCI standard "signatures" folder. We build with only a light make wrapper around docker today, though it assumes you have it configured to use the containerd image store backend, which allows for getting deterministic local digests without uploading to a registry. No reason you cannot build with podman or kaniko etc with some tweaks (which we hope to support officially) > Do you all document how you got around system level sources of non-determinism? Filesystems, metadata, timestamps, tempfiles, etc? This would be a great thing to document for people aiming for the same thing. We try to keep our package definitions to "FROM scratch" in "linux from scratch" style with no magic to be self documenting to be easy to audit or reference. By all means crib any of our tactics. We use no global env, so each package has only the determinism tweaks needed (if any). We heavily referenced Alpine, Arch, Mirage, Guix, Nix, and Debian to arrive at our current patterns. > What are you all using to verify commits? Are you guys verifying signatures against a public PKI? We all sign commits, reviews, and releases with well published PGP keys maintained in smartcards, with expected public keys in the MAINTAINERS file. Most of us have keyoxide profiles as well making it easy to prove all our online presences agree with the expected fingerprints for us. > Super interested as I manage the reproducibility program for a large software company. By all means drop in our matrix room, #stagex:matrix.org . Not many people working on these problems. The more we can all collaborate to unblock each other the better!

floxy 2y ago

>full source bootstrapped from 180 bytes of human-auditable machine code What does this mean? You have a C-like compiler in 180 bytes of assembler that can compile a C compiler that can then compile GCC?

mananaysiempre 2y ago
That’s normally what this means, yes, with a few more intermediate steps. There’s only one bootstrap chain like this that I know of[1,2,3], maintained by Jeremiah Orians and the Guix project; judging from the reference to 180 bytes, that’s what the distro GP describes is using as well. > This is a set of manually created hex programs in a Cthulhu Path to madness fashion. Which only have the goal of creating a bootstrapping path to a C compiler capable of compiling GCC, with only the explicit requirement of a single 1 KByte binary or less. [1] https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-building-from-source-all-the-way-down/ https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-... [2] https://savannah.nongnu.org/projects/stage0/ https://savannah.nongnu.org/projects/stage0/ [3] https://github.com/oriansj/bootstrap-seeds https://github.com/oriansj/bootstrap-seeds
- floxy 2y ago
  That's pretty awesome
  - lrvick 2y ago
    Yep, Guix and stagex are the only two distros that full source bootstrap to my knowleldge. We use an abbreviated and explicit stage0 chain here for easy auditing: https://codeberg.org/stagex/stagex/src/branch/main/packages/bootstrap/stage0/Containerfile https://codeberg.org/stagex/stagex/src/branch/main/packages/...
    - pabs3 2y ago
      IIRC the FreeDesktop flatpak runtimes are also built from the Bootstrappable Builds folks full source bootstrap.

deleted 2y ago

[deleted]