7 ms·

There are a lot of reasons to prefer reproducible builds, and many of them are not security related... It seems a bit presumptuous to argue that noone needs rep

There are a lot of reasons to prefer reproducible builds, and many of them are not security related... It seems a bit presumptuous to argue that noone needs reproducible builds because one particular security argument is flawed. First, a non-flawed security argument: it only takes one non-malicious person to build a package from source and find that it doesn't match the distributed binary to spot a problem. Sure, if you don't compile the binaries yourself, you might not find out until later that a binary was compromised, but that's still better than never finding out. The reality is that most people don't want to spend time building all their packages from source... More generally, reproducible builds make build artifacts a pure function of their inputs. There are countless reasons why this might be desirable. - If a binary is lost, it can be rebuilt exactly as it was. You only need to ensure the source is preserved. - If a particular version of the code is tested, and the binary is not a pure function of the code, then you haven't really tested the binary. Bugs could still be introduced that were not caught during testing because your build is non-deterministic. - It provides a foundation for your entire OS image to be built deterministically. - If you use a build cache, intermediate artifacts can be cached more easily, and use less space. For example, changing the code from A -> B -> A will result in two distinct artifacts instead of three.

vmchale 6y ago
Yes, CI is the big one for me.
maxerickson 6y ago
The link addresses 7 arguments.
arcticbull 6y ago
> - If a particular version of the code is tested, and the binary is not a pure function of the code, then you haven't really tested the binary. Bugs could still be introduced that were not caught during testing because your build is non-deterministic. This is a weaker argument IMO because when building for test, generally, all optimizations are disabled, debug info is emitted, symbols are un-stripped, and so on. The unit under test is usually very different from the shipped artifact even at the module level. Not least because the test functions are compiled in.
- wildmanx 6y ago
  If that's really what you are doing, then you are doing it horribly wrong.
  - arcticbull 6y ago
    You test prod builds?
    - joshuamorton 6y ago
      As part of the release process, yes...absolutely. Compared to basically every other part of release qualification (manual QA, canarying, etc.) re-testing on the prod build is so unbelievably cheap there's no reason to not.
      - arcticbull 6y ago
        I suppose we're referring to different kinds of testing. Manual QA, etc, on prod sure. But if you're building client software artifacts, to unit test or integration test involves building different software in a different configuration, with different software and running it in a test harness. To facilitate unit testing or integration testing client software you: - Build with a lower optimization level (-O0 usually) so that the generated code bares even a passing resemblance to what you actually wrote and your debugger can follow along. - Generate debug info. - Avoid stripping symbols. - Enable logging. - Build and link your tests code into a library artifact. - Run it in a test harness. That's not testing what you ship. It's testing something pretty close, obviously, but does not bear any semblance to a deterministic build.
        joshuamorton 6y ago
        No, I'm saying you re-run your automated unit tests on the release build because there's no reason not to. If you have test failures in opt/stripped mode, they're more annoying to debug yes, but wouldn't you want to know? Another way of putting this is that when you > - Build and link your tests code into a library artifact. You build and link the same object files that will be built into the release binary artifact, deterministically. > - Enable logging. I, uhh, usually do this in my released software too.
TazeTSchnitzel 6y ago
Reproducible builds make diagnosing performance regressions less horrible, because they are less likely to have been caused by random shuffling of code, and any random shuffling is reproducible when trying to bisect.
chacham15 6y ago
Reproducible builds also tend to create isolation. I.e. if I compile one C file, change another, the compilation of the second one usually doesnt affect the first. This is useful when trying to debug code. E.g. I can add print statements to callers to get more info surrounding a bug when in a non-reproducible state, the recompilation itself may have eliminated the bug.
mikepurvis 6y ago
Having stable BuildIDs has been important for me in being able to sanely manage a debug symbol archive where some binaries are periodically rebuilt on different CI worker nodes.
joshuamorton 6y ago
Note: This article is not talking about deterministic builds (which are a prerequisite for reproducible builds), but specifically reproducible builds. Reproducible builds are generally speaking interesting only from a security perspective, while deterministic builds have all sorts of useful infrastructural features which the author agrees are useful. And if you think I'm being pedantic, I'm using the official terminology from the reproducible builds site[0]. [0]: https://reproducible-builds.org/ https://reproducible-builds.org/
- GauntletWizard 6y ago
  You're semantically right, but also missing the point. The expense of getting to deterministic builds is large - You have to take great care in your build infrastructure and scripts. The benefits are also large, and worth it. Once you've gotten to deterministic builds, the expense of getting to reproducible builds is small; Typically days worth of work as opposed to months. The benefits are very different, but far from insignificant, and almost invaluable from a security perspective. If you're going to do deterministic builds, go for broke - Do reproducible builds.
  - Ericson2314 6y ago
    What is the difference?
    - tux1968 6y ago
      Deterministic build means every time you compile the same source you get the same executable. Reproducible build means that you specify and relay enough information to allow everyone else to reproduce your results for themselves in their own environment.
      - Ericson2314 6y ago
        Haha thanks. So one has hidden variables that happen to be locally constant, and the other is purged of hidden variables.
m463 6y ago
I think reproducible builds might be similar to locking a very-very-infrequently used data structure.
richardwhiuk 6y ago
Agree. In critical infrastructure, it's often essential to provide customers with a single fix. If you can't reproduce the build that they have, then you can't do that.
- hinkley 6y ago
  We have an internal tool that grovels through dependencies looking for first party code, and then pulls that thread all the way to hyperlinks to the ticket numbers and the commit diffs. So not only can we build another copy with the same third party code, when we do a bug fix you can validate that you in fact only got that one change into the bugfix. Even with infrastructure to do surgery on dependencies, it's quite possible to fat finger something and get too much or not enough. So we built a sanity tool that gives the engineering sign-off prior to validation a bit of gravitas.
hinkley 6y ago
Scientific method debugging as well. If the new version is busted, I want to rebuild the same output only changing the compiler, or package manager, or one of the library upgrades, and see what happens. If I can't reproduce last tuesday then how do I get back to Friday when shit wasn't on fire?
- hinkley 6y ago
  This also goes the other way too. Repeatability is daylight that removes our ability to delude ourselves that a new problem must be someone else's code. I don't want to police my coworkers. I just want to know that the infrastructure will hold my proverbial weight when I step on it, so I don't have to be afraid of side effects all day. I want to know what I can trust them to do, and work to expand that envelope. If people break stuff, I eventually want them to be able to figure it out, reproduce it, and fix it all under their own steam, and [believe] that it's the right thing to do. My automation strategy is a superset of a list I'm sure you've all heard already: Make it documented Make it work (automated) Make it right (trustworthy) Make it recommended Make it easy (may include fast) Make it mandatory Make it an HR problem About the time the tool is starting to get easy, you can start teasing people for not using it, but at some point it's expected behavior. More peer pressure from more sources. If they still like to cowboy, that speaks to trust, and they start getting delisted from new initiatives. If that still doesn't work (which, sadly, occasionally is the case), they are compartmentalized and shortlisted for the next reorg or layoff.
SkyBelow 6y ago
>If a binary is lost, it can be rebuilt exactly as it was. You only need to ensure the source is preserved. Wouldn't this depend upon the environment as well? Unless the build starts off by creating a build environment, but then we are half way to "To first make bread from scratch, create a universe...".
- remexre 6y ago
  If you run your builds in Docker, this is usually taken care of. If not, it seems like it'd be good practice to document exactly what build dependencies are used; I've had to track down a regression introduced by inlined code from a dependency when we weren't tracking build dependency versions, and the fact that a bug was introduced with no changes in the relevant section of the code was troublesome to say the least.
NovemberWhiskey 6y ago
This is exactly right. Having the build process be 'reproducible' i.e. deterministic means it is insulated from mysterious action-at-a-distance mechanisms that break things I want or need to be invariant within my CI/CD process.
biddlesby 6y ago
Completely agree! The article is blinkered to the security aspect. But imagine if compilers didn't create reproducible builds. Debugging would be a nightmare!
- renox 6y ago
  Uh? Is that sarcasm? Compilers don't produce reproducible builds. If you try to investigate a core dump using a binary recompiled from sources instead of the original binary, it's very likely you won't be able to analyze the core..
  - TwoBit 6y ago
    Since when don't compilers produce reproducible builds? We did that at my last workplace with appropriate MSVC compiler options. In any case, maybe parent is referring to using centralized debug symbols which can work for anybody in the org because their compilers all generate the same output.
  - nix23 6y ago
    >it's very likely you won't be able to analyze the core.. You know that's the reproducible part, exact same library's same compiler same options. >Compilers don't produce reproducible builds. Sure they do, just compile two times the same and compare its hash, would be terrible if compilers makes something else every-time.
    - giantrobot 6y ago
      By default you're not guaranteed the exact same output in two compiled binaries. There's a lot of variable bits[1] that make into binaries from C and C++. Different languages/compilers have different levels of variable bits. [1] https://blog.conan.io/2019/09/02/Deterministic-builds-with-C-C++.html https://blog.conan.io/2019/09/02/Deterministic-builds-with-C...
      - nix23 6y ago
        >Different languages/compilers have different levels of variable bits. No that's the point! Exact same environment same binary, that's reproducibility.
nhoughto 6y ago
Just the build caching is worth the price of entry. Gradle especially does a great job at this.
- cmckn 6y ago
  Tangential, but I actually switched back to Maven after using Gradle on a fairly large project for about a year. The incremental/cached builds of Gradle were awesome, but I found writing build.gradle files to be a bit too hacky. I can achieve everything I need to with some simple configuration of an existing Maven plugin, whereas with Gradle, it always felt like I was doing something super custom and fragile. I'd still love to set up a Gradle cache server, sounds so fancy! Random piece of Gradle magic, a generated BOM: https://github.com/grpc/grpc-java/blob/master/bom/build.gradle https://github.com/grpc/grpc-java/blob/master/bom/build.grad...

vmchale 6y ago

Yes, CI is the big one for me.

maxerickson 6y ago

The link addresses 7 arguments.

arcticbull 6y ago

> - If a particular version of the code is tested, and the binary is not a pure function of the code, then you haven't really tested the binary. Bugs could still be introduced that were not caught during testing because your build is non-deterministic. This is a weaker argument IMO because when building for test, generally, all optimizations are disabled, debug info is emitted, symbols are un-stripped, and so on. The unit under test is usually very different from the shipped artifact even at the module level. Not least because the test functions are compiled in.

wildmanx 6y ago
If that's really what you are doing, then you are doing it horribly wrong.
- arcticbull 6y ago
  You test prod builds?
  - joshuamorton 6y ago
    As part of the release process, yes...absolutely. Compared to basically every other part of release qualification (manual QA, canarying, etc.) re-testing on the prod build is so unbelievably cheap there's no reason to not.
    - arcticbull 6y ago
      I suppose we're referring to different kinds of testing. Manual QA, etc, on prod sure. But if you're building client software artifacts, to unit test or integration test involves building different software in a different configuration, with different software and running it in a test harness. To facilitate unit testing or integration testing client software you: - Build with a lower optimization level (-O0 usually) so that the generated code bares even a passing resemblance to what you actually wrote and your debugger can follow along. - Generate debug info. - Avoid stripping symbols. - Enable logging. - Build and link your tests code into a library artifact. - Run it in a test harness. That's not testing what you ship. It's testing something pretty close, obviously, but does not bear any semblance to a deterministic build.
      - joshuamorton 6y ago
        No, I'm saying you re-run your automated unit tests on the release build because there's no reason not to. If you have test failures in opt/stripped mode, they're more annoying to debug yes, but wouldn't you want to know? Another way of putting this is that when you > - Build and link your tests code into a library artifact. You build and link the same object files that will be built into the release binary artifact, deterministically. > - Enable logging. I, uhh, usually do this in my released software too.

TazeTSchnitzel 6y ago

Reproducible builds make diagnosing performance regressions less horrible, because they are less likely to have been caused by random shuffling of code, and any random shuffling is reproducible when trying to bisect.

chacham15 6y ago

Reproducible builds also tend to create isolation. I.e. if I compile one C file, change another, the compilation of the second one usually doesnt affect the first. This is useful when trying to debug code. E.g. I can add print statements to callers to get more info surrounding a bug when in a non-reproducible state, the recompilation itself may have eliminated the bug.

mikepurvis 6y ago

Having stable BuildIDs has been important for me in being able to sanely manage a debug symbol archive where some binaries are periodically rebuilt on different CI worker nodes.

joshuamorton 6y ago

Note: This article is not talking about deterministic builds (which are a prerequisite for reproducible builds), but specifically reproducible builds. Reproducible builds are generally speaking interesting only from a security perspective, while deterministic builds have all sorts of useful infrastructural features which the author agrees are useful. And if you think I'm being pedantic, I'm using the official terminology from the reproducible builds site[0]. [0]: https://reproducible-builds.org/ https://reproducible-builds.org/

GauntletWizard 6y ago
You're semantically right, but also missing the point. The expense of getting to deterministic builds is large - You have to take great care in your build infrastructure and scripts. The benefits are also large, and worth it. Once you've gotten to deterministic builds, the expense of getting to reproducible builds is small; Typically days worth of work as opposed to months. The benefits are very different, but far from insignificant, and almost invaluable from a security perspective. If you're going to do deterministic builds, go for broke - Do reproducible builds.
- Ericson2314 6y ago
  What is the difference?
  - tux1968 6y ago
    Deterministic build means every time you compile the same source you get the same executable. Reproducible build means that you specify and relay enough information to allow everyone else to reproduce your results for themselves in their own environment.
    - Ericson2314 6y ago
      Haha thanks. So one has hidden variables that happen to be locally constant, and the other is purged of hidden variables.

m463 6y ago

I think reproducible builds might be similar to locking a very-very-infrequently used data structure.

richardwhiuk 6y ago

Agree. In critical infrastructure, it's often essential to provide customers with a single fix. If you can't reproduce the build that they have, then you can't do that.

hinkley 6y ago
We have an internal tool that grovels through dependencies looking for first party code, and then pulls that thread all the way to hyperlinks to the ticket numbers and the commit diffs. So not only can we build another copy with the same third party code, when we do a bug fix you can validate that you in fact only got that one change into the bugfix. Even with infrastructure to do surgery on dependencies, it's quite possible to fat finger something and get too much or not enough. So we built a sanity tool that gives the engineering sign-off prior to validation a bit of gravitas.

hinkley 6y ago

Scientific method debugging as well. If the new version is busted, I want to rebuild the same output only changing the compiler, or package manager, or one of the library upgrades, and see what happens. If I can't reproduce last tuesday then how do I get back to Friday when shit wasn't on fire?

hinkley 6y ago
This also goes the other way too. Repeatability is daylight that removes our ability to delude ourselves that a new problem must be someone else's code. I don't want to police my coworkers. I just want to know that the infrastructure will hold my proverbial weight when I step on it, so I don't have to be afraid of side effects all day. I want to know what I can trust them to do, and work to expand that envelope. If people break stuff, I eventually want them to be able to figure it out, reproduce it, and fix it all under their own steam, and [believe] that it's the right thing to do. My automation strategy is a superset of a list I'm sure you've all heard already: Make it documented Make it work (automated) Make it right (trustworthy) Make it recommended Make it easy (may include fast) Make it mandatory Make it an HR problem About the time the tool is starting to get easy, you can start teasing people for not using it, but at some point it's expected behavior. More peer pressure from more sources. If they still like to cowboy, that speaks to trust, and they start getting delisted from new initiatives. If that still doesn't work (which, sadly, occasionally is the case), they are compartmentalized and shortlisted for the next reorg or layoff.

SkyBelow 6y ago

>If a binary is lost, it can be rebuilt exactly as it was. You only need to ensure the source is preserved. Wouldn't this depend upon the environment as well? Unless the build starts off by creating a build environment, but then we are half way to "To first make bread from scratch, create a universe...".

remexre 6y ago
If you run your builds in Docker, this is usually taken care of. If not, it seems like it'd be good practice to document exactly what build dependencies are used; I've had to track down a regression introduced by inlined code from a dependency when we weren't tracking build dependency versions, and the fact that a bug was introduced with no changes in the relevant section of the code was troublesome to say the least.

NovemberWhiskey 6y ago

This is exactly right. Having the build process be 'reproducible' i.e. deterministic means it is insulated from mysterious action-at-a-distance mechanisms that break things I want or need to be invariant within my CI/CD process.

biddlesby 6y ago

Completely agree! The article is blinkered to the security aspect. But imagine if compilers didn't create reproducible builds. Debugging would be a nightmare!

renox 6y ago
Uh? Is that sarcasm? Compilers don't produce reproducible builds. If you try to investigate a core dump using a binary recompiled from sources instead of the original binary, it's very likely you won't be able to analyze the core..
- TwoBit 6y ago
  Since when don't compilers produce reproducible builds? We did that at my last workplace with appropriate MSVC compiler options. In any case, maybe parent is referring to using centralized debug symbols which can work for anybody in the org because their compilers all generate the same output.
- nix23 6y ago
  >it's very likely you won't be able to analyze the core.. You know that's the reproducible part, exact same library's same compiler same options. >Compilers don't produce reproducible builds. Sure they do, just compile two times the same and compare its hash, would be terrible if compilers makes something else every-time.
  - giantrobot 6y ago
    By default you're not guaranteed the exact same output in two compiled binaries. There's a lot of variable bits[1] that make into binaries from C and C++. Different languages/compilers have different levels of variable bits. [1] https://blog.conan.io/2019/09/02/Deterministic-builds-with-C-C++.html https://blog.conan.io/2019/09/02/Deterministic-builds-with-C...
    - nix23 6y ago
      >Different languages/compilers have different levels of variable bits. No that's the point! Exact same environment same binary, that's reproducibility.

nhoughto 6y ago

Just the build caching is worth the price of entry. Gradle especially does a great job at this.

cmckn 6y ago
Tangential, but I actually switched back to Maven after using Gradle on a fairly large project for about a year. The incremental/cached builds of Gradle were awesome, but I found writing build.gradle files to be a bit too hacky. I can achieve everything I need to with some simple configuration of an existing Maven plugin, whereas with Gradle, it always felt like I was doing something super custom and fragile. I'd still love to set up a Gradle cache server, sounds so fancy! Random piece of Gradle magic, a generated BOM: https://github.com/grpc/grpc-java/blob/master/bom/build.gradle https://github.com/grpc/grpc-java/blob/master/bom/build.grad...