6 ms·
Reproducible Builds in January 2022
- smashed 5y agoThanks for sharing, did not know this project existed. I am maintaining a small embedded linux project myself and having a reproducible build as a feature is a long standing goal. The more tooling and awareness there is, the better.
- throwaway984393 5y agoIt's amazing to me how much the open source community accomplishes that huge corporations with billions of dollars can't even get close to. Thanks to all the open source contributors who give their spare time to make improvements for everybody.
- rapsey 5y agoA huge amount of open source projects are done by people payed to develop them.
- goodpoint 5y agoyet the Reproducible Builds project has been started and driven almost entirely by Debian volunteers: https://reproducible-builds.org/who/people/ https://reproducible-builds.org/who/people/
- cowsandmilk 5y agoThat page specifically says their work is funded. I’m sure all the core members do a ton of volunteer work for Debian, but I believe they are now being paid for work on reproducible builds.
- sanxiyn 5y agoIt's funded by donation to Reproducible Builds project, and donation is managed by Software Freedom Conservancy (free software organization which also manages donation for Git, QEMU, Samba, and Wine). It's not like they are paid by their employers. You can donate yourself. It is tax-deductible in US. They include logos for donation larger than $25,000. Logos are here: https://reproducible-builds.org/who/sponsors/ https://reproducible-builds.org/who/sponsors/. Top sponsor is Google Open Source Security Team.
- goodpoint 5y agoNo, funded does not mean that people are paid for every hour of contribution they do or that they do it full-time. Of the entire set of contributors, most are paid zero. Some are paid.
- IshKebab 5y agoI don't know if that's fair. One of the biggest advance in build systems for ages is Bazel and that came from Google.
- dub 5y agoIt's no accident that the Google Open Source Security Team is sponsor of reproducible-builds.org: they'd like to get the open source world up to speed with best practices that have been applied widely internally at Google for well over a decade. Blaze (a.k.a. Bazel) rolled out at Google around 2007-ish, and the idea that consistent inputs should produce consistent outputs was fundamental to its philosophy. Build rules like https://github.com/GoogleContainerTools/distroless https://github.com/GoogleContainerTools/distroless to create minimal, reproducible docker images would seem radical and new most people building docker containers these days (almost everyone uses Dockerfile format) but it'd seem perfectly ordinary and very old-fashioned to any Googler.
- dijit 5y agoI feel like I’ve hit something really special with Bazel, but there’s almost no ecosystem around it. Im trying to use it for game development, the model fits so perfectly for large monorepo projects with explicit dependencies, but it seems like since there’s no ecosystem that I have to write a lot of my own rules from scratch; and the learning resources are many years old and in the conference format. It’s quite dense to get in. Even the slack requires an @google.com email to get in.
- SmileyKeith 5y ago> Even the slack requires an @google.com email to get in. This shouldn't be the case, I'm in this slack and it's mostly non-google folks. Do you see this at https://slack.bazel.build https://slack.bazel.build ?
- dijit 5y agoI did that page was a log in or sign up page, which required an @google.com email to sign up: someone has fixed it now, it wasn’t working on Friday which was the last time I checked.
- kristjansson 5y agoThat seemed shocking, but it doesn’t appear to be true? https://github.com/bazelbuild/rules_swift/issues/536#issuecomment-1030448400 https://github.com/bazelbuild/rules_swift/issues/536#issueco...
- smartineng 5y agoIts a pitty that Linux itself still cannot be bootstrapped without the Linux env - at least no one has done it yet. Anyway thanks to this reproducible-builds efforts maybe the awarness of this serious problem will grow and we will see more support into the projects like https://github.com/fosslinux/live-bootstrap https://github.com/fosslinux/live-bootstrap , https://niedzejkob.p4.team/bootstrap/ https://niedzejkob.p4.team/bootstrap/
- emidln 5y agoI admit I've never tried this, but the Linux kernel seems like just another C program. What's to stop me from installing GNU Make, GNU coreutils, gcc, and binutils cross compiled to target x86_64-linux-elf and bootstrap this? It doesn't sound especially useful, but it does sound like something that would be possible if you focused on it.
- sanxiyn 5y agoYes, it should work. You aren't in any disagreement actually. It's just that no one has done it yet, but should have.
- stickac 5y agoNixOS reproducibility effort is tracked here: https://r13y.com https://r13y.com
- tazjin 5y agoIt is very ironic that the only non-reproducible path in the minimal channel is Nix itself (> 2.3)
- georgyo 5y agoI like and use nixos as a primary driver, but the nix tool itself needs a product manager. A lot of people have direct commit access and it is really hard to figure out near and medium term goals or their progress. It's how the tool that does so much to ensure other things built are reproducible is not reproducible itself.
- dec0dedab0de 5y agoCould someone with more knowledge explain why this is so difficult? As I understand it, the goal should be that if you download a binary, and download the source and compile a binary, they should be identical. I understand different build flags, and compilers. But if a project documents their build system why wouldnt it be reproducable?
- aaron42net 5y agoBy default, many compilers include things like local filesystem paths, build server hostnames, or build timestamps into their binary artifacts. These will obviously differ build-to-build. Even without that, it's possible to accidentally leak entropy into the build output. For example, readdir() doesn't guarantee any kind of ordering, so without sorting the list of files it is possible for a binary artifact (or even tar) to produce different output from the same input.
- dmm 5y agoThe build environment seems to sneak into the artifacts in many ways. Check out some of the listings from the "Upstream Patches" section of the linked page.. Some of the most common reasons seem to be: timestamps, non-deterministic file listings, debug information containing build path data, permissions bits on artifact files,etc.
- icegreentea2 5y agoDepending on the tools used, it can be "easy" for things in the environment to leak into the build. Timestamps are a common bane of reproducible builds. I understand that a non-trivial amount of reproducible build work consists of hunting down every timestamp in a build, and then finding a way to inject alternate timestamps into them. For example if we just look at one of the listed reproducibility bugs in the report (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864082 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864082) you can get a flavor of the random bits and bods that need to be hunted down. We could say that a system can be relatively easily made reproducible if starting from scratch - certainly all of the tooling is there. The far more significant challenge is retrofitting reproducibility into existing large projects.