4 ms·

It is very cool to see distros working together for a common goal. But I still do not understand the point of "reproducible builds". I know what they are, but

• david-gpu 2y ago
  It's a safety measure. Reproducible builds ensure identical binaries are produced from the same source. They help detect e.g. hidden backdoors.
• samsartor 2y ago
  The video gets into that. The main purpose is to verify that the binary you're running came from the actual source code.
• ssivark 2y ago
  What makes you so confident that the benefit is less than the effort? Given the increasing likelihood of supply chain attacks, isn’t this a very prudent precaution?
• 3s 2y ago
  A really important application of reproducible builds is running code inside Secure Enclaves that has been committed to on a public transparency log. A client can connect to a remote secure enclave that can then prove to the client that it’s running the commit code via a process known as remote attestation. It’s pretty cool stuff. However it’s only possible if the build inside the enclave is reproducible (deterministic) and always identical to the build on the transparency log
• NegativeLatency 2y ago
  Would make stuff like this harder to pull off: https://en.wikipedia.org/wiki/XZ_Utils_backdoor https://en.wikipedia.org/wiki/XZ_Utils_backdoor
  • nindalf 2y ago
    Are you sure? If I'm understanding correctly, the malicious code was introduced as part of the test code, so no matter who compiled it, they'd get a binary with the same (malicious) functionality. Heck, it might even have been reproducibly malicious. The real crazy part was that it was modifying the functionality of sshd at runtime, allowing the attacker to log into any system. Reproducibility of either sshd or xz wouldn't have stopped this attack. That's my reading of https://research.swtch.com/xz-script https://research.swtch.com/xz-script.
    • solarkraft 2y ago
      I tend to agree. The exploit was (by detour) committed to the source.
      • ramses0 2y ago
        Technically, interestingly, if `bazel` were used as the build tool, it would avoid the straightforward ability to cross-contaminate the build executable with the test code... Yeah, `bazel run test:...` would have access to the test files, but `bazel build xz:executable` would not (by default) be able to pull in extra shenanigans from the test files (and I think there's generally linting and formatting rules required by default with `BUILD.bazel` files, reducing another sneak-vectors)
  • kpcyrd 2y ago
    It unfortunately doesn't help in cases like this. Reproducible Builds gives you a trusted path from source to binary, but it doesn't help with backdoors in the source code/build instructions. For that we'd need some sort of source code reviewing effort like https://github.com/crev-dev/cargo-crev https://github.com/crev-dev/cargo-crev implements. I've started whatsrc.org to keep track of the source code inputs we're putting into our computers (that would benefit from reviews), but the conclusion is also somewhat "it's too much".
• champtar 2y ago
  With reproducible build you know that what you test on your dev laptop is the same as what will go out from your CI, and if hash mismatch you can chase why. For a concrete exemple, Mellanox driver configure script will auto detect if it's running under docker and change a compile flags, so if you build in a container using podman you get a different result.
• solarkraft 2y ago
  > but to me the amount of work involved outweighs the benefit I don’t know whether I’d spend this much work on such an abstract goal, but what reproducibility changes really is quite amazing. It vastly increases trust in published binaries and obviates the need for signing and the security benefit of compiling software yourself.
  • gruez 2y ago
    >obviates the need for signing and the security benefit of compiling software yourself Not really. Most people still would rely on signatures because they can't be expected to compile everything from scratch just to verify their download is authentic. Moreover even though reproducible builds make verification easier, it still requires someone to sound the alarm. For less popular packages there might be nobody checking any particular build is backdoored, because most people see "reproducible builds" and they assume Somebody Else is doing the reproduction.
    • mjl- 2y ago
      for transparency of reproducible builds of go applications, i made https://beta.gobuilds.org/ https://beta.gobuilds.org/. it compiles any publicly available go application on-demand, with a toolchain version of your choice (latest stable by default), for a platform of your choice. all (pure) go applications are reproducible by default, including when cross-compiled, and go toolchains run nothing provided by the go module (awesome properties!). the source code is verified through the go sum database (a transparency log containing go modules). the hash of the resulting binary is added to gobuild's own transparency log. so it can be publicly verified. the gobuilds service builds the binary itself, and has another instance (on a different platform & config) build the binary too, to ensure the binary is really reproducible (i'ld like other instances that i don't run myself as secondaries too). i no longer publish binaries for my applications (that i write in go). i just point to the "latest"-build link for the go module at gobuilds. also makes it easy for users (including myself) to get new builds for new go toolchains (which may include fixes to the (relatively large, and often used) standard library). you still may not trust the public gobuilds instance. my hope is that people (eg software projects themselves, or distros, or other kinds of communities) will run & use their own gobuild instances and verify their builds against the public gobuilds service. win-win: gives them assurance their builds are really reproducible, and builds trust in the public gobuilds (keeping it honest, if someone sees a hash mismatch, they will speak up). i usually don't get much enthusiasm for it though. (:
• noirscape 2y ago
  Practically speaking, the idea with a reproducible build is that you can take the source files they used, run their instructions the same way they did and get hash for hash[0] the specific resulting executable. The main benefit is that you can trust that the resulting binary file being served matches the source code that it's build from. This mostly matters for distros in that they build from a source package repository, but anyone running a mirror could hypothetically replace the package with another (potentially malicious) package, leading users to install malicious tooling. It mainly matters for distros because pretty much every distro out there runs on third party mirrors (often ran by universities, but also just people who want to help) rather than on direct upstream; packages get uploaded to a main server, then mirrors copy from that main server (to reduce network traffic load on the main server). Right now, mirror trust is mostly "we assume you're not gonna be evil, until we get complaints". If the build is reproducible, the software can inherently confirm that the file they're getting is trustworthy, making "getting complaints" much easier to confirm. It can also speed up the overall building process; if the package source code hasn't changed, you can also always assume that the resulting binary hasn't changed (meaning you can use hashes instead of relying on mtime like make does). Docker build cache works in a somewhat similar way (although docker isn't inherently deterministic). Devwise, you can also reconstruct a build much easier if it's reproducible; ie. if you've accidentally thrown away the .elf file for debugging, if your build is deterministic, you can just rerun the build and get the same .elf file again. [0]: While not a problem for Linux distros, in cases where you need a secret to sign an application, reproducible typically means "identical except for the signature" instead. F-Droid uses this for example to figure out if they should use buildserver stuff or the original APKs: https://f-droid.org/docs/Reproducible_Builds/ https://f-droid.org/docs/Reproducible_Builds/
  • yellow_lead 2y ago
    > a mirror could hypothetically replace the package with another (potentially malicious) package, leading users to install malicious tooling. It was my assumption that a mirror is required to host a build that has a hash conforming to the original. Is that not the case?
    • 2y ago
• jcranmer 2y ago
  The main benefit you'll hear touted is something along the lines of being able to get an attestation that the resulting artifact was built following the steps claimed to build it. I think that's a somewhat overstated benefit, though, as it's not clear to me that this is an avenue of attack used in practice, given the frequency with which software already has vulnerabilities usable for exploits, or the ease with which one can insert a backdoor into the source code (e.g., the xz backdoor). I think the actual main utility is that the process has done a very good job of rooting out several causes of unintentional nondeterminism in the build process. I say unintentional because the two main causes of unreproducibility, by several orders of magnitude, are timestamps being embedded everywhere and absolute paths being embedded everywhere, and those are rather expected. But some of the unreproducibility comes from things like accidental reliance on inodes in file paths (i.e., doing "for file in listdir()" without sorting the results of listdir) or the compiler itself accidentally sorting based on pointer address (which is unreproducible on ASLR systems).
  • uecker 2y ago
    The xz backdoor was news because they went to a lot of effort to try to hide a backdoor in the source (actually a binary file in the source) and still failed. In contrast, without reproducible builds it is trivial for a maintainer with upload rights (or somebody who managed to get the credentials from a maintainer) to insert a backdoor into a binary. And it is then virtually impossible to detect.
    • rcxdude 2y ago
      Well, the xz backdoor was detected through the behaviour of the resulting binaries, not through observation of the source code tampering, so I don't think it's a great example.
      • drowsspa 2y ago
        Yeah, I would say the x backdoor succeeded summa cum laude
      • uecker 2y ago
• zelphirkalt 2y ago
  It is also about reliably being able to built things a month from now, in a year, in 5 years, etc.