8 ms·

Fully Countering Trusting Trust Through Diverse Double-Compiling (2009)

88 points by micaeked 10y ago

jacques_chester 10y ago
The hard part -- sometimes really hard -- is the "bit-for-bit identical" requirement. Lots of builds are recreatable but not reproducible (there is probably a better term of art here). You can go back to a point in time and build the version of the software as it was, but you are not guaranteed to get a bit-for-bit clone. (See https://reproducible-builds.org https://reproducible-builds.org for a thorough discussion) The problem is that there are lots of uncontrolled inputs to a build that are due to sourcecode or compiler changes. Most famously there are timestamps and random numbers, which mess up all sorts of hashing-based approaches. These can even be non-obvious. Just the other day I and a colleague were investigating the (small but unsettling) possibility that an old buildpack had been replaced maliciously. We compared the historical hash to the file: different. We rebuilt the historical buildpack with trusted inputs: still different. Then we unzipped both versions and diff'd the directories: identical. What had thrown our hashes off was that zipfiles, by default, include timestamps. We have a build that is recreatable but not reproducible. Speaking of builds, we are able to reproducibly build some binaries but not others. Off the top of my head our most high-profile non-reproducible build is NodeJS. Some other binaries (Ruby and Python, in my not-at-all-complete recollection) are fully reproducible. This difficulty with fully reproducing makes it hard to provide a fully trustworthy chain of custody. A company which uses Cloud Foundry have in actual fact stood up an independent copy of our build pipelines inside their own secure network, so that they can be completely autarkic for the build steps leading to a complete buildpack. This doesn't defend against malicious source, but it defends against malicious builds. Disclosure: I work for Pivotal, the majority donor of engineering to Cloud Foundry. As you've probably guessed, I'm currently a fulltime contributor on the buildpacks team.
- sly010 10y ago
  Nixos goes to a great length to steer towards reproduceability. They run builds in chroots, they set all ctime to past 0, they make all directories except the output directory read only, etc. But even then compilers have all sorts of quirks, like running certain code paths multiple times and deciding which one runs faster on ~this~ CPU. The biggest conceptual mistake we are making is that by default compilers always build for ~this~ machine, linking to this libraries. This makes it so the state of the machine inherently changes with every compilation (aka compiling is not a purely functional operation anymore). If I could go back time and change automake and glibc, cross compiling and explicit dependency handling should be the norm. (As an aside, containers would greatly benefit too as you wouldn't need to package an entire linux distribution with every binary) I am sometimes amazed, sometimes disappointed by this reproduceability problem. Computers supposed to be machines that can do the same thing again and again without a mistake, but this is not the case anymore. We have so many layers of complexity and everything is bolted together with duct tape. We focus on developer convenience in the short term but in the long term we completely loose determinism. Sure we can write more code faster than before, but building software is more problematic than ever. Yet, somehow everything seems to be going to this direction, in fact some people celebrate it and compare it to biology or evolution. I just call is "accidentally stochastic computing".
  - srtjstjsj 10y ago
    > some people celebrate it and compare it to biology or evolution. Creating life is scientifically exhilarating, but incredibly dangerous.
- mjquinn 10y ago
  This is a problem that the GNU Guix package manager[0] (and presumably its inspiration Nix[1]) are helping to solve. Any two git checkouts of Guix with the same git hash on the same architecture should produce bit-identical builds across time and space for many of the programs it packages. It's not true for everything they package yet, but they're making progress. (Some might find the documentation for the `guix challenge` command interesting: https://www.gnu.org/software/guix/manual/html_node/Invoking-guix-challenge.html https://www.gnu.org/software/guix/manual/html_node/Invoking-...) [0] https://www.gnu.org/software/guix/ https://www.gnu.org/software/guix/ [1] https://nixos.org/nix/ https://nixos.org/nix/
- predakanga 10y ago
  Interestingly, this problem appears in some pretty diverse circumstances. The one that springs to mind is video game emulation, from more than a decade ago - the communities there needed a means to reproducibly compress their ROMs, both for verifying dumps and for sharing large sets of ROMs. The tools created have been through many iterations, but they're still in use today in the form of TorrentZip[0] and various relatives like torrent7z. [0]: https://github.com/uwedeportivo/torrentzip https://github.com/uwedeportivo/torrentzip
  - jacques_chester 10y ago
    Well I know what tool I'll be looking at on Monday :)
- zzzcpan 10y ago
  There are some efforts [1] to make reproducible builds really work, also nix guys have some experience with them, as others have noted. Isolated deterministic environments and stripping binaries/archives (strip-nondeterminism tool) [2] generally do the trick. [1] https://reproducible-builds.org https://reproducible-builds.org [2] https://reproducible-builds.org/tools/ https://reproducible-builds.org/tools/
  - jacques_chester 10y ago
    Some of my predecessors on buildpacks went through a bunch of work to establish reproducibility for binaries we ship, with varied levels of success: "Investigate how we can allow users to independently verify/authenticate a final buildpack" (https://www.pivotaltracker.com/story/show/104469634 https://www.pivotaltracker.com/story/show/104469634) "Explore: Compiled binaries should be reproducible" (https://www.pivotaltracker.com/story/show/104746074 https://www.pivotaltracker.com/story/show/104746074) "determine whether the libfaketime reproducible build strategy will work across all of our binaries" (https://www.pivotaltracker.com/story/show/107752798 https://www.pivotaltracker.com/story/show/107752798) "Investigate Why are our node builds not reproducible?" (https://www.pivotaltracker.com/story/show/128161137 https://www.pivotaltracker.com/story/show/128161137) As well as supporting work to help independent verification of the "chain of custody". There's 25 of those under that label, if you use the search box.
- skybrian 10y ago
  Bit-for-bit reproducibility does require changing how you do things. Always use checksums, not dates. For zip we have a special version that sets all the timestamps in the archive to a fixed value. Anything writing records to a file based on a hash table needs to sort the entries. The team needs to be dedicated to making all its tools work this way. The good part is that you have a clear goal.
- mynameislegion 10y ago
  Next time, use diffoscope to track down the exact differences.
  - jacques_chester 10y ago
    Thanks for the tip!
dwheeler 10y ago
I'm the author. Ask me anything about it.
- adekok 10y ago
  What are the implications if we don't start from a trusted compiler? e.g. we have compilers A and B. Either of which may, or may not, have back doors. Is it possible to detect back doors via compilation? A complex backdoor can detect compilation of A or B, and insert different backdoor code into each. But will those back doors be identical? i.e. is there a a compilation path (A -> B' -> B'', etc.) such that we can either detect that the backdoors are either bit for bit identical, or that there are no back doors?
  - dwheeler 10y ago
    Awesome question, and there's a sneaky answer. DDC works if the trusted compiler has back doors and other nastiness - as long as the back doors won't affect the DDC results. This is noted in section 4.3: "... something is “trusted” if we have justified confidence that it does not have triggers and payloads that would affect the results of DDC. A trusted program or process may have triggers and payloads, as long as they do not affect the result. A trusted program or process may have defects, though as shown later, any defects that affect its result in DDC are likely to be detected. Methods to increase the level of confidence are discussed in chapter 6." http://www.dwheeler.com/trusting-trust/dissertation/html/wheeler-trusting-trust-ddc.html#4.3.Informal%20assumptions http://www.dwheeler.com/trusting-trust/dissertation/html/whe... Chapter 6 discusses various ways to make this likely: http://www.dwheeler.com/trusting-trust/dissertation/html/wheeler-trusting-trust-ddc.html#6.Methods%20to%20increase%20diversity http://www.dwheeler.com/trusting-trust/dissertation/html/whe... While you're doing tests, it's probably best to do them on an isolated system or network. One interesting appraoch is to use old computers to compile newer compilers (possibly through emulation). It's not likely that the older computers will have malicious attacks or backdoors that will work against newer compilers. You can also apply it multiple times using multiple different trusted compilers. In that case, an attack would have had to subvert all of those compilers (and/or their environment). This quickly becomes vanishingly unlikely.
dwheeler 10y ago
jacques_chester: You're absolutely right that recreating things bit-for-bit identical can require some real elbow grease. I had to overcome bugs in the Tiny C compiler (tcc) and gcc to make them work, as described in the paper. Date/time stamps can create problems too. But all these problems are quite doable. Nobody claims that gcc is small, yet I managed to get that working. Compiler makers can follow a few guidelines to make it much easier, see: http://www.dwheeler.com/trusting-trust/dissertation/html/wheeler-trusting-trust-ddc.html#4.Guidelines%20for%20Compiler%20Suppliers http://www.dwheeler.com/trusting-trust/dissertation/html/whe... Check out the graph at the Debian Reproducible Builds project at https://wiki.debian.org/ReproducibleBuilds https://wiki.debian.org/ReproducibleBuilds - yes, some packages don't reproduce bit-for-bit, but they've made a tremendous amount of progress and managed it for most packages. You can see some related information at this page: http://www.dwheeler.com/trusting-trust/ http://www.dwheeler.com/trusting-trust/ including a video of me discussing the paper.
- jacques_chester 10y ago
  Do you have any tips on encouraging upstream projects to invest in reproducible builds? Our ideal world would be fully reproducible builds with a complete chain of custody. We have some of it, but not the whole kit and kaboodle. But we can't really do this so long as we rely on unreproducible upstream build configurations.
  - dwheeler 10y ago
    I think at least part of the solution is convincing upstreams that the world has changed. There are now many people and organizations who are actively working to subvert software - and some of them have a lot of resources and incentive. They're working to break into a variety of things (including repositories, build systems, and distribution processes) so that people run subverted software. Frankly, the world changed decades ago, but only recently have many developers started to realize it. I try to convince people by pointing out past attacks, for example. One piece that might help you is: https://blog.torproject.org/blog/deterministic-builds-part-one-cyberwar-and-global-compromise https://blog.torproject.org/blog/deterministic-builds-part-o... Of course, you then have to convince them what specifically to do. The reproducible builds project has some nice documentation: https://reproducible-builds.org/docs/ https://reproducible-builds.org/docs/ and I already mentioned my guidelines: http://www.dwheeler.com/trusting-trust/dissertation/html/wheeler-trusting-trust-ddc.html#4.Guidelines%20for%20Compiler%20Suppliers http://www.dwheeler.com/trusting-trust/dissertation/html/whe... . You can also look at specific war stories, such as Tor's: https://blog.torproject.org/blog/deterministic-builds-part-two-technical-details https://blog.torproject.org/blog/deterministic-builds-part-t... or sbcl's: http://christophe.rhodes.io/notes/blog/posts/2014/reproducible_builds_-_a_month_ahead_of_schedule/ http://christophe.rhodes.io/notes/blog/posts/2014/reproducib... We can also make it easier. One great thing is that the Debian reproducible builds group has been modifying tools to make it easier to create reproducible builds. That doesn't mean there's nothing left to do, but making it easier makes it way more likely. The "containerization of everything" also has the potential to make life easier - it makes it easier to start from some fixed point, and repeat a sequence of instructions from there.
rurban 10y ago
Can we please add a (2009) to the title? This is a classic paper on reproducible builds, everybody is working on since. Better overview: http://www.dwheeler.com/trusting-trust/ http://www.dwheeler.com/trusting-trust/ Older discussion, 7 years ago: * https://news.ycombinator.com/item?id=1104338 https://news.ycombinator.com/item?id=1104338
nickpsecurity 10y ago
This again. A perfect example of solving the wrong problems in a clever way. To his credit, Wheeler at least gives credit to the brilliant engineer (Karger) who invented the attack, points out it took 10 years before that knowledge reached anyone via Thompson (recurring problem in high-security), and did the reference essays on the two solutions to the actual problem (high-assurance FLOSS & SCM's). That's what you're better off reading. Here's a quick enumeration of the problems in case people wonder why I gripe about this and reproducible builds fad: 1. What the compiler does needs to be fully specified and correct to ensure security. 2. The implementation of it in the language should conform to that spec or simply be correct itself. 3. No backdoors are in the compiler, the compilation process, etc. This must be easy to show. 4. The optimizations used don't break security/correctness. 5. The compiler can parse malicious input without code injection resulting. 6. The compilation of the compiler itself follows all of the above. 7. The resulting binary that everyone has is the same one matching the source with same correct or malicious function but no malicious stuff added that's not in the source code already. This equivalence is what everyone in mainstream is focusing on. I already made an exception for Wheeler himself given he did this and root cause work. 8. The resulting binary will then be used on systems developed without mitigating problems above to compile other apps not mitigating problems above. So, that's a big pile of problems. The Thompson attack, countering the Thompson attack, or reproducible builds collectively address the tiniest problem vs all the problems people actually encounter with compilers and compiler distribution. There's teams working on the latter that have produced nice solutions to a bunch of them. VLISP, FLINT, the assembly-to-LISP-to-HLL project & CakeML-to-ASM come to mind. There's commercial products, like CompCert, available as well. Very little by mainstream in FOSS or proprietary. The "easy" approach to solve most of the real problem is a certifying compiler in a safe language bootstrapped on a simple, local one whose source is distributed via secure SCM. In this case, you do not have a reproducible build in vast majority of cases since you've verified source itself and have a verifying compiler to ASM. You'll even benefit from no binary where your compiler can optimize the source for your machine or even add extra security to it (a la Softbound+CETS). Alternatively, you can get the binary that everyone can check via signatures on the secure SCM. You can even do reproducible builds on top of my scheme for the added assurance you get in reproducing bugs or correctness of specific compilations. Core assurance... 80/20 rule... comes from doing a compiler that's correct-by-construction much as possible, easy for humans to review for backdoors, and on secure repo & distribution system. Meanwhile, the big problems are ignored and these little, tactical solutions to smaller problems keep getting lots of attention. Same thing that happen between Karger and Thompson time frame for Karger et al's other recommendations for building secure systems. We saw where that went in terms of the baseline of INFOSEC we had for decades. ;) Note: I can provide links on request to definitive works on subversion, SCM, compiler correctness, whatever. I think the summary in this comment should be clear. Hopefully. Note 2: Anyone that doubts I'm right can try an empirical approach of looking at bugs, vulnerabilities and compromises published for both GCC and things compiled with it. Look for number of times they said, "We were owned by the damned Thompson attack. If only we countered it with diverse, double compilation or reproducible builds." Compare that to failures in other areas on my list. How unimportant this stuff is vs higher-priority criteria should be self-evident at that point. And empirically proven.
contingencies 10y ago
It strikes me that the use of diverse systems to reinforce assumptions of trust within a given subsystem is an architectural paradigm not limited to compilers. The key problems are implementation feature-set or edge-case differences and overhead (real time and maintenance/up-front development). In fact, it would be ideal with multiple client versions/implementations on any service (particularly distributed or financial) and indeed I have done this in the past. Not sure if this paradigm has a name... anyone? I suppose you could just say consensus-based hedging.
- nickpsecurity 10y ago
  It's in fact a long-established technique in high-assurance systems going back to I think aerospace or security-critical where the triple-modular redundancy trick was reapplied with separate teams building each one. Security through diversity also re-emerged relatively recently as a very active sub-field of INFOSEC/IT. If you're interested in that stuff, use these terms in various combination when you're Googling: "artificial diversity," "automated obfuscation," "moving target software security," "security diversity software." Also, including "pdf" helps given most good ones are papers. The word "survey" will occasionally land you on a pile of gold with references all in one spot. :) Happy hunting!
bitwize 10y ago
I still can't see this paper referenced without thinking of HISSATSU! DOUBLE COMPILE!: https://www.youtube.com/watch?v=FHkFzRZdlV4 https://www.youtube.com/watch?v=FHkFzRZdlV4
lamby 10y ago
Hi all, I do a lot of work on Reproducible Builds within Debian (AMA...!). Just wanted to mention we are now having regular IRC meetings: https://lists.reproducible-builds.org/pipermail/rb-general/2016-October/000071.html https://lists.reproducible-builds.org/pipermail/rb-general/2...

88 points by micaeked 10y ago

The hard part -- sometimes really hard -- is the "bit-for-bit identical" requirement. Lots of builds are recreatable but not reproducible (there is probably a better term of art here). You can go back to a point in time and build the version of the software as it was, but you are not guaranteed to get a bit-for-bit clone. (See https://reproducible-builds.org https://reproducible-builds.org for a thorough discussion) The problem is that there are lots of uncontrolled inputs to a build that are due to sourcecode or compiler changes. Most famously there are timestamps and random numbers, which mess up all sorts of hashing-based approaches. These can even be non-obvious. Just the other day I and a colleague were investigating the (small but unsettling) possibility that an old buildpack had been replaced maliciously. We compared the historical hash to the file: different. We rebuilt the historical buildpack with trusted inputs: still different. Then we unzipped both versions and diff'd the directories: identical. What had thrown our hashes off was that zipfiles, by default, include timestamps. We have a build that is recreatable but not reproducible. Speaking of builds, we are able to reproducibly build some binaries but not others. Off the top of my head our most high-profile non-reproducible build is NodeJS. Some other binaries (Ruby and Python, in my not-at-all-complete recollection) are fully reproducible. This difficulty with fully reproducing makes it hard to provide a fully trustworthy chain of custody. A company which uses Cloud Foundry have in actual fact stood up an independent copy of our build pipelines inside their own secure network, so that they can be completely autarkic for the build steps leading to a complete buildpack. This doesn't defend against malicious source, but it defends against malicious builds. Disclosure: I work for Pivotal, the majority donor of engineering to Cloud Foundry. As you've probably guessed, I'm currently a fulltime contributor on the buildpacks team.

sly010 10y ago
Nixos goes to a great length to steer towards reproduceability. They run builds in chroots, they set all ctime to past 0, they make all directories except the output directory read only, etc. But even then compilers have all sorts of quirks, like running certain code paths multiple times and deciding which one runs faster on ~this~ CPU. The biggest conceptual mistake we are making is that by default compilers always build for ~this~ machine, linking to this libraries. This makes it so the state of the machine inherently changes with every compilation (aka compiling is not a purely functional operation anymore). If I could go back time and change automake and glibc, cross compiling and explicit dependency handling should be the norm. (As an aside, containers would greatly benefit too as you wouldn't need to package an entire linux distribution with every binary) I am sometimes amazed, sometimes disappointed by this reproduceability problem. Computers supposed to be machines that can do the same thing again and again without a mistake, but this is not the case anymore. We have so many layers of complexity and everything is bolted together with duct tape. We focus on developer convenience in the short term but in the long term we completely loose determinism. Sure we can write more code faster than before, but building software is more problematic than ever. Yet, somehow everything seems to be going to this direction, in fact some people celebrate it and compare it to biology or evolution. I just call is "accidentally stochastic computing".
- srtjstjsj 10y ago
  > some people celebrate it and compare it to biology or evolution. Creating life is scientifically exhilarating, but incredibly dangerous.
mjquinn 10y ago
This is a problem that the GNU Guix package manager[0] (and presumably its inspiration Nix[1]) are helping to solve. Any two git checkouts of Guix with the same git hash on the same architecture should produce bit-identical builds across time and space for many of the programs it packages. It's not true for everything they package yet, but they're making progress. (Some might find the documentation for the `guix challenge` command interesting: https://www.gnu.org/software/guix/manual/html_node/Invoking-guix-challenge.html https://www.gnu.org/software/guix/manual/html_node/Invoking-...) [0] https://www.gnu.org/software/guix/ https://www.gnu.org/software/guix/ [1] https://nixos.org/nix/ https://nixos.org/nix/
predakanga 10y ago
Interestingly, this problem appears in some pretty diverse circumstances. The one that springs to mind is video game emulation, from more than a decade ago - the communities there needed a means to reproducibly compress their ROMs, both for verifying dumps and for sharing large sets of ROMs. The tools created have been through many iterations, but they're still in use today in the form of TorrentZip[0] and various relatives like torrent7z. [0]: https://github.com/uwedeportivo/torrentzip https://github.com/uwedeportivo/torrentzip
- jacques_chester 10y ago
  Well I know what tool I'll be looking at on Monday :)
zzzcpan 10y ago
There are some efforts [1] to make reproducible builds really work, also nix guys have some experience with them, as others have noted. Isolated deterministic environments and stripping binaries/archives (strip-nondeterminism tool) [2] generally do the trick. [1] https://reproducible-builds.org https://reproducible-builds.org [2] https://reproducible-builds.org/tools/ https://reproducible-builds.org/tools/
- jacques_chester 10y ago
  Some of my predecessors on buildpacks went through a bunch of work to establish reproducibility for binaries we ship, with varied levels of success: "Investigate how we can allow users to independently verify/authenticate a final buildpack" (https://www.pivotaltracker.com/story/show/104469634 https://www.pivotaltracker.com/story/show/104469634) "Explore: Compiled binaries should be reproducible" (https://www.pivotaltracker.com/story/show/104746074 https://www.pivotaltracker.com/story/show/104746074) "determine whether the libfaketime reproducible build strategy will work across all of our binaries" (https://www.pivotaltracker.com/story/show/107752798 https://www.pivotaltracker.com/story/show/107752798) "Investigate Why are our node builds not reproducible?" (https://www.pivotaltracker.com/story/show/128161137 https://www.pivotaltracker.com/story/show/128161137) As well as supporting work to help independent verification of the "chain of custody". There's 25 of those under that label, if you use the search box.
skybrian 10y ago
Bit-for-bit reproducibility does require changing how you do things. Always use checksums, not dates. For zip we have a special version that sets all the timestamps in the archive to a fixed value. Anything writing records to a file based on a hash table needs to sort the entries. The team needs to be dedicated to making all its tools work this way. The good part is that you have a clear goal.
mynameislegion 10y ago
Next time, use diffoscope to track down the exact differences.
- jacques_chester 10y ago
  Thanks for the tip!

dwheeler 10y ago

I'm the author. Ask me anything about it.

adekok 10y ago
What are the implications if we don't start from a trusted compiler? e.g. we have compilers A and B. Either of which may, or may not, have back doors. Is it possible to detect back doors via compilation? A complex backdoor can detect compilation of A or B, and insert different backdoor code into each. But will those back doors be identical? i.e. is there a a compilation path (A -> B' -> B'', etc.) such that we can either detect that the backdoors are either bit for bit identical, or that there are no back doors?
- dwheeler 10y ago
  Awesome question, and there's a sneaky answer. DDC works if the trusted compiler has back doors and other nastiness - as long as the back doors won't affect the DDC results. This is noted in section 4.3: "... something is “trusted” if we have justified confidence that it does not have triggers and payloads that would affect the results of DDC. A trusted program or process may have triggers and payloads, as long as they do not affect the result. A trusted program or process may have defects, though as shown later, any defects that affect its result in DDC are likely to be detected. Methods to increase the level of confidence are discussed in chapter 6." http://www.dwheeler.com/trusting-trust/dissertation/html/wheeler-trusting-trust-ddc.html#4.3.Informal%20assumptions http://www.dwheeler.com/trusting-trust/dissertation/html/whe... Chapter 6 discusses various ways to make this likely: http://www.dwheeler.com/trusting-trust/dissertation/html/wheeler-trusting-trust-ddc.html#6.Methods%20to%20increase%20diversity http://www.dwheeler.com/trusting-trust/dissertation/html/whe... While you're doing tests, it's probably best to do them on an isolated system or network. One interesting appraoch is to use old computers to compile newer compilers (possibly through emulation). It's not likely that the older computers will have malicious attacks or backdoors that will work against newer compilers. You can also apply it multiple times using multiple different trusted compilers. In that case, an attack would have had to subvert all of those compilers (and/or their environment). This quickly becomes vanishingly unlikely.

dwheeler 10y ago

jacques_chester: You're absolutely right that recreating things bit-for-bit identical can require some real elbow grease. I had to overcome bugs in the Tiny C compiler (tcc) and gcc to make them work, as described in the paper. Date/time stamps can create problems too. But all these problems are quite doable. Nobody claims that gcc is small, yet I managed to get that working. Compiler makers can follow a few guidelines to make it much easier, see: http://www.dwheeler.com/trusting-trust/dissertation/html/wheeler-trusting-trust-ddc.html#4.Guidelines%20for%20Compiler%20Suppliers http://www.dwheeler.com/trusting-trust/dissertation/html/whe... Check out the graph at the Debian Reproducible Builds project at https://wiki.debian.org/ReproducibleBuilds https://wiki.debian.org/ReproducibleBuilds - yes, some packages don't reproduce bit-for-bit, but they've made a tremendous amount of progress and managed it for most packages. You can see some related information at this page: http://www.dwheeler.com/trusting-trust/ http://www.dwheeler.com/trusting-trust/ including a video of me discussing the paper.

jacques_chester 10y ago
Do you have any tips on encouraging upstream projects to invest in reproducible builds? Our ideal world would be fully reproducible builds with a complete chain of custody. We have some of it, but not the whole kit and kaboodle. But we can't really do this so long as we rely on unreproducible upstream build configurations.
- dwheeler 10y ago
  I think at least part of the solution is convincing upstreams that the world has changed. There are now many people and organizations who are actively working to subvert software - and some of them have a lot of resources and incentive. They're working to break into a variety of things (including repositories, build systems, and distribution processes) so that people run subverted software. Frankly, the world changed decades ago, but only recently have many developers started to realize it. I try to convince people by pointing out past attacks, for example. One piece that might help you is: https://blog.torproject.org/blog/deterministic-builds-part-one-cyberwar-and-global-compromise https://blog.torproject.org/blog/deterministic-builds-part-o... Of course, you then have to convince them what specifically to do. The reproducible builds project has some nice documentation: https://reproducible-builds.org/docs/ https://reproducible-builds.org/docs/ and I already mentioned my guidelines: http://www.dwheeler.com/trusting-trust/dissertation/html/wheeler-trusting-trust-ddc.html#4.Guidelines%20for%20Compiler%20Suppliers http://www.dwheeler.com/trusting-trust/dissertation/html/whe... . You can also look at specific war stories, such as Tor's: https://blog.torproject.org/blog/deterministic-builds-part-two-technical-details https://blog.torproject.org/blog/deterministic-builds-part-t... or sbcl's: http://christophe.rhodes.io/notes/blog/posts/2014/reproducible_builds_-_a_month_ahead_of_schedule/ http://christophe.rhodes.io/notes/blog/posts/2014/reproducib... We can also make it easier. One great thing is that the Debian reproducible builds group has been modifying tools to make it easier to create reproducible builds. That doesn't mean there's nothing left to do, but making it easier makes it way more likely. The "containerization of everything" also has the potential to make life easier - it makes it easier to start from some fixed point, and repeat a sequence of instructions from there.

rurban 10y ago

Can we please add a (2009) to the title? This is a classic paper on reproducible builds, everybody is working on since. Better overview: http://www.dwheeler.com/trusting-trust/ http://www.dwheeler.com/trusting-trust/ Older discussion, 7 years ago: * https://news.ycombinator.com/item?id=1104338 https://news.ycombinator.com/item?id=1104338

nickpsecurity 10y ago

This again. A perfect example of solving the wrong problems in a clever way. To his credit, Wheeler at least gives credit to the brilliant engineer (Karger) who invented the attack, points out it took 10 years before that knowledge reached anyone via Thompson (recurring problem in high-security), and did the reference essays on the two solutions to the actual problem (high-assurance FLOSS & SCM's). That's what you're better off reading. Here's a quick enumeration of the problems in case people wonder why I gripe about this and reproducible builds fad: 1. What the compiler does needs to be fully specified and correct to ensure security. 2. The implementation of it in the language should conform to that spec or simply be correct itself. 3. No backdoors are in the compiler, the compilation process, etc. This must be easy to show. 4. The optimizations used don't break security/correctness. 5. The compiler can parse malicious input without code injection resulting. 6. The compilation of the compiler itself follows all of the above. 7. The resulting binary that everyone has is the same one matching the source with same correct or malicious function but no malicious stuff added that's not in the source code already. This equivalence is what everyone in mainstream is focusing on. I already made an exception for Wheeler himself given he did this and root cause work. 8. The resulting binary will then be used on systems developed without mitigating problems above to compile other apps not mitigating problems above. So, that's a big pile of problems. The Thompson attack, countering the Thompson attack, or reproducible builds collectively address the tiniest problem vs all the problems people actually encounter with compilers and compiler distribution. There's teams working on the latter that have produced nice solutions to a bunch of them. VLISP, FLINT, the assembly-to-LISP-to-HLL project & CakeML-to-ASM come to mind. There's commercial products, like CompCert, available as well. Very little by mainstream in FOSS or proprietary. The "easy" approach to solve most of the real problem is a certifying compiler in a safe language bootstrapped on a simple, local one whose source is distributed via secure SCM. In this case, you do not have a reproducible build in vast majority of cases since you've verified source itself and have a verifying compiler to ASM. You'll even benefit from no binary where your compiler can optimize the source for your machine or even add extra security to it (a la Softbound+CETS). Alternatively, you can get the binary that everyone can check via signatures on the secure SCM. You can even do reproducible builds on top of my scheme for the added assurance you get in reproducing bugs or correctness of specific compilations. Core assurance... 80/20 rule... comes from doing a compiler that's correct-by-construction much as possible, easy for humans to review for backdoors, and on secure repo & distribution system. Meanwhile, the big problems are ignored and these little, tactical solutions to smaller problems keep getting lots of attention. Same thing that happen between Karger and Thompson time frame for Karger et al's other recommendations for building secure systems. We saw where that went in terms of the baseline of INFOSEC we had for decades. ;) Note: I can provide links on request to definitive works on subversion, SCM, compiler correctness, whatever. I think the summary in this comment should be clear. Hopefully. Note 2: Anyone that doubts I'm right can try an empirical approach of looking at bugs, vulnerabilities and compromises published for both GCC and things compiled with it. Look for number of times they said, "We were owned by the damned Thompson attack. If only we countered it with diverse, double compilation or reproducible builds." Compare that to failures in other areas on my list. How unimportant this stuff is vs higher-priority criteria should be self-evident at that point. And empirically proven.

contingencies 10y ago

It strikes me that the use of diverse systems to reinforce assumptions of trust within a given subsystem is an architectural paradigm not limited to compilers. The key problems are implementation feature-set or edge-case differences and overhead (real time and maintenance/up-front development). In fact, it would be ideal with multiple client versions/implementations on any service (particularly distributed or financial) and indeed I have done this in the past. Not sure if this paradigm has a name... anyone? I suppose you could just say consensus-based hedging.

nickpsecurity 10y ago
It's in fact a long-established technique in high-assurance systems going back to I think aerospace or security-critical where the triple-modular redundancy trick was reapplied with separate teams building each one. Security through diversity also re-emerged relatively recently as a very active sub-field of INFOSEC/IT. If you're interested in that stuff, use these terms in various combination when you're Googling: "artificial diversity," "automated obfuscation," "moving target software security," "security diversity software." Also, including "pdf" helps given most good ones are papers. The word "survey" will occasionally land you on a pile of gold with references all in one spot. :) Happy hunting!

bitwize 10y ago

I still can't see this paper referenced without thinking of HISSATSU! DOUBLE COMPILE!: https://www.youtube.com/watch?v=FHkFzRZdlV4 https://www.youtube.com/watch?v=FHkFzRZdlV4

lamby 10y ago

Hi all, I do a lot of work on Reproducible Builds within Debian (AMA...!). Just wanted to mention we are now having regular IRC meetings: https://lists.reproducible-builds.org/pipermail/rb-general/2016-October/000071.html https://lists.reproducible-builds.org/pipermail/rb-general/2...