8 ms·
Status update from the Reproducible Builds project
- gtt 9y agoHow do they achieve reproducibility with python and some other languages which include timestamps and such?
- anonacct37 9y agohttps://reproducible-builds.org/docs/timestamps/ https://reproducible-builds.org/docs/timestamps/
- JoshTriplett 9y agoSometimes by eliminating the timestamps (don't put __DATE__ and __TIME__ in C code, for instance), but when that isn't possible: https://reproducible-builds.org/specs/source-date-epoch/ https://reproducible-builds.org/specs/source-date-epoch/
- mapreri 9y agoAlso, Debian binary packages don't ship prebuilt python bytecode, which is compiled at installation time. (just to reply to the specific example)
- rnhmjoj 9y agoOn NixOS python is patched so that if the environment variable DETERMINISTIC_BUILD is present the interpreter set the bytecode timestamps to 0. I suppose they did something similar.
- seagreen 9y agoAmazing work. Thanks so much to everyone who's contributing. The upstream bugs filed are especially appreciated since they make the whole Linux ecosystem more solid, not just Debian.
- jbergstroem 9y agoNot just Linux; FreeBSD and NetBSD have been along the ride for a while: https://reproducible-builds.org/who/ https://reproducible-builds.org/who/
- protomyth 9y agowith NetBSD stating "As of 2017-02-20 we have fully reproducible builds on amd64 and sparc64." https://blog.netbsd.org/tnf/entry/netbsd_fully_reproducible_builds https://blog.netbsd.org/tnf/entry/netbsd_fully_reproducible_...
- deleted 9y ago[deleted]
- seagreen 9y agoWhoops, good point. Didn't mean to be so Linux-centric.
- cperciva 9y agoIn some cases, a very long while. I brought up the question of build reproducibility at BSDCon 2003 because it was relevant to FreeBSD Update, and a lot of my early FreeBSD commits were working on this.
- jbergstroem 9y agoYeah; I've seen both work and developer mindset about this for a long time in bsd-centric mailing-lists. I tried to keep it short here though, seeing how the debian developers have done a great job and didn't want to shift that focus [in this thread]. I think its great that we have come to a point where packagers shift mindset from "it works" to "we can reproduce the results" in more than one package manager.
- cperciva 9y agoDoes anyone know if they've made the Packages file (repository metadata file, listing the packages in the repository) build reproducibly yet? I tripped over this a couple weeks ago and was both amused and annoyed, since it seemed that packages were being listed in the file in a random order. I'm asking here because it might already be fixed; we're using a slightly old version of the package/repository tools.
- phreack 9y agoIn case anyone is not aware of what reproducibility is and why it's a worthy goal, here's their statement: https://wiki.debian.org/ReproducibleBuilds/About https://wiki.debian.org/ReproducibleBuilds/About
- pen2l 9y agoWhat does "reproducibility" mean? I understand and appreciate the importance of reproducibility in the context of scientific experiments, but I don't understand what it means in terms of computer programs. I am guessing it has to do with being able to build on different architectures without issue?
- richdougherty 9y agoI think it's for security. It means that there's a deterministic relationship between the source of a program and its final compiled artifacts. If software has reproducible builds that means that third-parties can independently verify that artifacts have been built safely from sources, without any sneaky stuff snuck in.
- CJefferson 9y agoNo, the same architecture. It has taken, and continues to take, a suprising amount of work to make two builds of a program produce the same output. There are many sources of issues. For example: date and time stored in output, program's output effected by order in which files are read from a directory (and is not having a fixed ordering), hash tables based on the pointer and the high objects are stored having different ordering on different executions, parallel programs behaving differently on different runs, and others.
- jldugger 9y agoIt's about being able to reproduce the binary from source. You might think this is pretty much impossible in the Debian context, but things like timestamps, and underspecified dependencies can end up shifting a build's result over time. If we want to insist that open source code is secure by source code analysis, we need a verifiable build chain, that the code and binaries an analysis uses are the same as what we get later.
- cyphar 9y agoIt has a similar meaning to research. What it means is that you can reproduce (compile in most cases) from source code the same bit-for-bit identical binary independently. While this might sound like something that should be trivial to do, it turns out to be far from trivial (timestamps and other environment information leaks into binaries all the time). There's a website that describes this project in much more detail as well as how they worked around the various problems they found. https://reproducible-builds.org/ https://reproducible-builds.org/
- dingdingdang 9y agoSo happy someone are spending time on this issue, it's like a breath of fresh air and intelligence in the midst of all the usual software (security/privacy/etc. take your pick) mayhem. It's worth reading https://reproducible-builds.org/ https://reproducible-builds.org/ for a brief (re-)reminder on why this project is important. Outtake: "With reproducible builds, multiple parties can redo this process independently and ensure they all get exactly the same result. We can thus gain confidence that a distributed binary code is indeed coming from a given source code."
- Kenji 9y agoReproducible builds are extremely useful. There are more benefits. For example, suppose you have a build server compiling software packages. If your builds are not reproducible and you want to debug a core dump, but you have no debug information, you are out of luck (well, you could dive into the assembly code, but it's inconvenient). If you want to keep debug information, you need to store them for every single build (what a waste of storage...) because the binary for each build is different. Not so with reproducible builds, you could simply check out the old version and compile it with debug information!
- mschuster91 9y agoMost huge Debian packages carry a separate -debug package so you can get the symbols without huge recompilation times + having to set up a buildchain and associated dev libraries of all the packages.
- jakeogh 9y agoSimilarly Gentoo has 'splitdebug': https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Backtraces#Stripping https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Backt...
- deleted 9y ago[deleted]
- pmoriarty 9y agoHow does the kind of reproducibility spoken of here compare to that offered by Guix and Nix?
- arianvanp 9y agoGuix and Nix are input-reproducible. Given the same input description (input being the source files and any dependencies) an output comes out. Builds are then looked up in a build cache based on the hash of al lathe combined inputs. However. The _output_ of Nix artifacts are not reproducible. Running the same input twice will yield a different result. Nix does some tricks to improve output reproducibility like building things in sandboxes with fixed time, and using tarballs without modification dates but output bit-by-bit reproducible is not their goal. They also don't have the manpower for this. Currently, a build is built by a trusted builderver for which you have the public key. And you look up the built by input hash but have no way to check if the thing the builderver is serving is legit. It's fully based on trust. However, with debian putting so much effort in reproducible output, Nix can benefit too. In the future, we would like to get rid of the 'trust-based' build servers and instead move to a consensus model. Say if 3 servers give the same output hash given an input hash, then we trust that download and avoid a compile from source. If you still don't trust it, you can build from source yourself and check if the source is trustworthy. Summary: Nix does not do bit-by-bit reproducibility, but we benefit greatly from the work that debian is doing. In the future we will look at setting up infrastructure for buildservers with an output-hash based trust model instead of an input based one. However this will take time. Feel Free to ask any other questions.
- rekado 9y ago> output bit-by-bit reproducible is not their goal I think you are wrong. The Nix people (and the Guix people, including myself) are also involved in the reproducible builds project. I've met with a couple of them in Berlin last year. It's not just Debian doing this. I can't speak for Nix but for the Guix project bit-for-bit reproducibility is an explicitly stated goal. It's very important and the reason why Guix is used in several HPC environments as the foundation for reproducible research. Disclaimer: I'm co-maintainer of GNU Guix and currently at a bioinfo conference where I talked about Guix and reproducibility.
- adamb 9y agoBeyond providing security, reproducible builds also provide an important ingredient for caching build artifacts (and thus accelerating build times) across CI and developer machines. They also can form the basis of a much simpler deploy and update pipeline, where the version of source code deployed is no longer as important. Instead a simple (recursive) binary diff can identify which components of a system must be updated, and which have not changed since the last deploy. This means a simpler state machine with fewer edge cases that works more quickly and reliably than the alternative. I'm very grateful for the work that this project has done and continues to do. Thank you!
- morecoffee 9y agoOnce we have reproducible builds, will it be possible to have verifiable builds? As in, can we cryptographically show that source + compiler = binary? Right now we can sign source code, we can sign binaries, but we can't shows that source produced binaries. I would feel much happier about installing code if I knew it was from a particular source or author.
- detaro 9y agoWhat do you mean by "cryptographically show"? With reproducible builds, anyone that has repeated a build can verify that a claimed binary matches, and could sign a statement saying so. But I don't think there are solutions that don't include someone repeating the build, or a clear way of proving that you actually did repeat the build.
- zcdziura 9y agoJust curious: how does one do that? Hash both the binary and source code files, that THAT hash, then sign it? Or am I completely off?
- tedunangst 9y agoNo. Or at least this doesn't provide that. I think in theory you could make a crypto compiler that proves the binary is isogonal to the source, but I suspect the verification effort wouldn't be much less than recompiling.
- nickpsecurity 9y agoYes. The first standard for securing computer systems mandated some protections against this. They were partly made by Paul Karger who invented the compiler subversion Thompson wrote about a decade later. Most just focused on that one thing where Karger et al went on to build systems that were secure from ground up with some surviving NSA pentesting and analysis for 2-5 years. Independently, people started building verified compilers and whole stacks w/ CPU's. They were initially simple with a lot to trust but got better over time. Recently, the two schools have been merging more. Mainstream INFOSEC and IT just ignores it all slowly reinventing it piece by piece with knock offs. It's hard, has performance hit, or is built in something other than C language so don't do it. (shrugs) Here's several examples: VLISP for Scheme48 whose papers are here: https://en.wikipedia.org/wiki/PreScheme https://en.wikipedia.org/wiki/PreScheme C0 compiler + whole stack correctness in Verisoft http://www.verisoft.de/VerisoftRepository.html http://www.verisoft.de/VerisoftRepository.html CompCert Compiler for C http://compcert.inria.fr/ http://compcert.inria.fr/ CakeML Subset of Standard ML https://cakeml.org/ https://cakeml.org/ Rockwell-Collins doing crypto DSL compiled to verified CPU http://www.ccs.neu.edu/home/pete/acl206/slides/hardin.pdf http://www.ccs.neu.edu/home/pete/acl206/slides/hardin.pdf Karger's original paper with the attack from 1970's: https://www.acsac.org/2002/papers/classic-multics.pdf https://www.acsac.org/2002/papers/classic-multics.pdf Myer's landmark work on subversion in high-assurance security from 1980: http://csrc.nist.gov/publications/history/myer80.pdf http://csrc.nist.gov/publications/history/myer80.pdf My framework I developed studying Karger back when I was building secure things: http://pastebin.com/y3PufJ0V http://pastebin.com/y3PufJ0V
- mabbo 9y agoOne (misguided) counter argument I've heard from otherwise fantastic devs it's the notion of adding randomness to unit tests in the hopes that if there's a bug, at least some builds will fail. In practice, I've seen those builds and developers saying "yeah, sometimes you need to build it twice". I think the solution is to give those devs who favor such techniques a separate but easy to use fuzzing tool set that they can run just like their unit tests, separate from their usual 'build' command. Give them their ability to discover new bugs, but make it separate from the real build.
- sitkack 9y agoNon-determinism should be on a leaf of the DAG.
- closeparen 9y agoWhy would the randomness in unit tests affect the binary? RNGs are invoked when the tests are run, not when they're built, and anyway, test code shouldn't be part of the final binary.
- davexunit 9y agoThe test suite is used to validate the build. Intermittently failing test suite -> intermittently failing build. You can always disable running a problematic test suite, but that doesn't exactly inspire confidence in the result.
- regularfry 9y agoMy take on that is that those devs are probably fooling themselves about the state of their test suite. I'm working on a codebase like that right now - it's in generally a relatively good state, but just this week I've run into a moderately nasty bug involving global state not getting properly reset which the tests didn't cover because their specific execution order happened to work.
- kobeya 9y agoDown-voted because that's not an argument against reproducible builds. The outputs can be bit-for-bit identical but still use an RNG for fuzz testing.
- Cogito 9y agoHas anyone played with the tool they mentioned, diffoscope? Sounds interesting and wonder how good it is at, for example, comparing excel files with VBA code and formulas etc.
- Sir_Cmpwn 9y agoCompare this to Windows or OSX, where not only are you unable to build packages yourself, but they are installed from downloads you find in disparate places on the web, are not cryptographically signed by people you can trust, and often include spyware anyway.