4 ms·

Could someone with more knowledge explain why this is so difficult? As I understand it, the goal should be that if you download a binary, and download the sour

Could someone with more knowledge explain why this is so difficult? As I understand it, the goal should be that if you download a binary, and download the source and compile a binary, they should be identical. I understand different build flags, and compilers. But if a project documents their build system why wouldnt it be reproducable?

aaron42net 5y ago
By default, many compilers include things like local filesystem paths, build server hostnames, or build timestamps into their binary artifacts. These will obviously differ build-to-build. Even without that, it's possible to accidentally leak entropy into the build output. For example, readdir() doesn't guarantee any kind of ordering, so without sorting the list of files it is possible for a binary artifact (or even tar) to produce different output from the same input.
dmm 5y ago
The build environment seems to sneak into the artifacts in many ways. Check out some of the listings from the "Upstream Patches" section of the linked page.. Some of the most common reasons seem to be: timestamps, non-deterministic file listings, debug information containing build path data, permissions bits on artifact files,etc.
icegreentea2 5y ago
Depending on the tools used, it can be "easy" for things in the environment to leak into the build. Timestamps are a common bane of reproducible builds. I understand that a non-trivial amount of reproducible build work consists of hunting down every timestamp in a build, and then finding a way to inject alternate timestamps into them. For example if we just look at one of the listed reproducibility bugs in the report (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864082 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864082) you can get a flavor of the random bits and bods that need to be hunted down. We could say that a system can be relatively easily made reproducible if starting from scratch - certainly all of the tooling is there. The far more significant challenge is retrofitting reproducibility into existing large projects.
- k__ 5y ago
  Why are people using timestamps in builds and what is the better alternative?
  - capableweb 5y ago
    Some people want to know when something was built. Better option is to use Git commit of build or even last-modified of the file it's being embedded into. Alternatively, allow a env var to override what date/time the software thinks it is.
  - richbell 5y ago
    > Why are people using timestamps in builds and what is the better alternative? Timestamps are often an implicit part of the build which leads to the same code producing 'different' artifacts (e.g. file created timestamp, generated metadata about the build stored in the artifact itself). One alternative is to support explicitly defining the 'build timestamp' so all builds with the same timestamp produces the same artifact. This is what Maven does[0]; I'd imagine it's much harder for build systems that download and recompile dependencies. [0] https://maven.apache.org/guides/mini/guide-reproducible-builds.html https://maven.apache.org/guides/mini/guide-reproducible-buil...
  - michaelt 5y ago
    First of all, every filesystem has timestamps and they've got to be set to some value. Second of all, it's useful to have some sort of version marker that lets you tell at a glance if two copies of the software are the same, or what code was running when that log file was created. Build timestamps are a classic way of doing this in a version-control-system-independent way, without advancing the 'headline' version number for every developer's test builds. And of course, if you include things like debug symbols it's useful to be able to tell if the source files on disk have changed since the build took place, as the line numbers could be all off. Alternatives include using a git commit hash; having a separate build pipeline for public release versions then not worrying if developers' builds aren't reproducible; and just being careful with developers' test builds.
marcosdumay 5y ago
Compilers are huge pieces of code, modified by lots and lots of hands, and up to recently nobody paid any attention if the results were identical or not. Your question is kinda like "if I write a few hundred lines of code, and run it, it should run, right?". The answer is yes, that's the expectation, but there is no chance that it is actually fulfilled in practice. As an example, when the Debian's reproducible builds project started, gcc had extra code to tag the executables with the time of compilation.
sanxiyn 5y ago
https://lists.reproducible-builds.org/pipermail/rb-general/2022-January/002472.html https://lists.reproducible-builds.org/pipermail/rb-general/2... should give you a better idea of what's going on. It's linked in the article. Perl sorting issues in dictionaries-common libxmlb used a pointer address (%p) for a hash value texlive-base: Reported differences in the generated ls-R
xurukefi 5y ago
Apart from what others have said, signed binaries also pose somewhat of a challenge with reproducible builds.
TacticalCoder 5y ago
In addition to various issues already mentioned, there are also some build systems that do build in parallel and produce builds or build artefacts that are going to depend on the order in which threads finished, which may vary from run to run. I'm not saying you cannot build in parallel and end up with a deterministic build: that is done all the time. What I'm saying in that there are cases where some care has to be taken.
jcranmer 5y ago
There are several things that can make builds nondeterministic: * Many file formats, especially (but not exclusively) archives like .zip, embed timestamps in them. * Absolute paths are also ridiculously easy to get embedded in the build artifact. * Other components of the user's environment can get embedded, including current locale, username, hostname. Some of these are more common than others. * Using an actual random number generator is more common than you might think (e.g., generating a cryptographic key during the build process), although still pretty rare. * Saying 'for file in directory' is usually sorted in inode order by default, which makes it inconsistent on different people's drives (although deterministic for a single user, generally). * Pointer addresses are nondeterministic. And it's easy to accidentally make a container that sorts by pointer address (e.g., std::map in C++). * In general, there's several features of programming languages that make it easy to accidentally inject nondeterminism in your program. Concurrency is obvious, but using standard containers like rust's HashMap or C++'s std::map combined with a "for every value in container" make it easy to get that nondeterminism. Compilers are not immune from being nondeterministic, and while it is a bug for them to be nondeterministic, it still happens.
- deleted 5y ago
  [deleted]
- noahtallen 5y ago
  Great examples! Another one is the version number, which is really helpful in lots of situations. It can also be useful to embed the git commit for debugging. (One practical use case for these is cache busting to ensure that a browser downloads a new version of a script, for example.)
- adonovan 5y ago
  Another surprising example: by default, some compilers attempt more aggressive optimization if there is ample memory available. (We discovered many surprising causes of nondeterminism during the development of Blaze aka Bazel, which prides itself on reproducibility.)

aaron42net 5y ago

By default, many compilers include things like local filesystem paths, build server hostnames, or build timestamps into their binary artifacts. These will obviously differ build-to-build. Even without that, it's possible to accidentally leak entropy into the build output. For example, readdir() doesn't guarantee any kind of ordering, so without sorting the list of files it is possible for a binary artifact (or even tar) to produce different output from the same input.

dmm 5y ago

The build environment seems to sneak into the artifacts in many ways. Check out some of the listings from the "Upstream Patches" section of the linked page.. Some of the most common reasons seem to be: timestamps, non-deterministic file listings, debug information containing build path data, permissions bits on artifact files,etc.

icegreentea2 5y ago

Depending on the tools used, it can be "easy" for things in the environment to leak into the build. Timestamps are a common bane of reproducible builds. I understand that a non-trivial amount of reproducible build work consists of hunting down every timestamp in a build, and then finding a way to inject alternate timestamps into them. For example if we just look at one of the listed reproducibility bugs in the report (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864082 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864082) you can get a flavor of the random bits and bods that need to be hunted down. We could say that a system can be relatively easily made reproducible if starting from scratch - certainly all of the tooling is there. The far more significant challenge is retrofitting reproducibility into existing large projects.

k__ 5y ago
Why are people using timestamps in builds and what is the better alternative?
- capableweb 5y ago
  Some people want to know when something was built. Better option is to use Git commit of build or even last-modified of the file it's being embedded into. Alternatively, allow a env var to override what date/time the software thinks it is.
- richbell 5y ago
  > Why are people using timestamps in builds and what is the better alternative? Timestamps are often an implicit part of the build which leads to the same code producing 'different' artifacts (e.g. file created timestamp, generated metadata about the build stored in the artifact itself). One alternative is to support explicitly defining the 'build timestamp' so all builds with the same timestamp produces the same artifact. This is what Maven does[0]; I'd imagine it's much harder for build systems that download and recompile dependencies. [0] https://maven.apache.org/guides/mini/guide-reproducible-builds.html https://maven.apache.org/guides/mini/guide-reproducible-buil...
- michaelt 5y ago
  First of all, every filesystem has timestamps and they've got to be set to some value. Second of all, it's useful to have some sort of version marker that lets you tell at a glance if two copies of the software are the same, or what code was running when that log file was created. Build timestamps are a classic way of doing this in a version-control-system-independent way, without advancing the 'headline' version number for every developer's test builds. And of course, if you include things like debug symbols it's useful to be able to tell if the source files on disk have changed since the build took place, as the line numbers could be all off. Alternatives include using a git commit hash; having a separate build pipeline for public release versions then not worrying if developers' builds aren't reproducible; and just being careful with developers' test builds.

marcosdumay 5y ago

Compilers are huge pieces of code, modified by lots and lots of hands, and up to recently nobody paid any attention if the results were identical or not. Your question is kinda like "if I write a few hundred lines of code, and run it, it should run, right?". The answer is yes, that's the expectation, but there is no chance that it is actually fulfilled in practice. As an example, when the Debian's reproducible builds project started, gcc had extra code to tag the executables with the time of compilation.

sanxiyn 5y ago

https://lists.reproducible-builds.org/pipermail/rb-general/2022-January/002472.html https://lists.reproducible-builds.org/pipermail/rb-general/2... should give you a better idea of what's going on. It's linked in the article. Perl sorting issues in dictionaries-common libxmlb used a pointer address (%p) for a hash value texlive-base: Reported differences in the generated ls-R

xurukefi 5y ago

Apart from what others have said, signed binaries also pose somewhat of a challenge with reproducible builds.

TacticalCoder 5y ago

In addition to various issues already mentioned, there are also some build systems that do build in parallel and produce builds or build artefacts that are going to depend on the order in which threads finished, which may vary from run to run. I'm not saying you cannot build in parallel and end up with a deterministic build: that is done all the time. What I'm saying in that there are cases where some care has to be taken.

jcranmer 5y ago

There are several things that can make builds nondeterministic: * Many file formats, especially (but not exclusively) archives like .zip, embed timestamps in them. * Absolute paths are also ridiculously easy to get embedded in the build artifact. * Other components of the user's environment can get embedded, including current locale, username, hostname. Some of these are more common than others. * Using an actual random number generator is more common than you might think (e.g., generating a cryptographic key during the build process), although still pretty rare. * Saying 'for file in directory' is usually sorted in inode order by default, which makes it inconsistent on different people's drives (although deterministic for a single user, generally). * Pointer addresses are nondeterministic. And it's easy to accidentally make a container that sorts by pointer address (e.g., std::map in C++). * In general, there's several features of programming languages that make it easy to accidentally inject nondeterminism in your program. Concurrency is obvious, but using standard containers like rust's HashMap or C++'s std::map combined with a "for every value in container" make it easy to get that nondeterminism. Compilers are not immune from being nondeterministic, and while it is a bug for them to be nondeterministic, it still happens.

deleted 5y ago
[deleted]
noahtallen 5y ago
Great examples! Another one is the version number, which is really helpful in lots of situations. It can also be useful to embed the git commit for debugging. (One practical use case for these is cache busting to ensure that a browser downloads a new version of a script, for example.)
adonovan 5y ago
Another surprising example: by default, some compilers attempt more aggressive optimization if there is ample memory available. (We discovered many surprising causes of nondeterminism during the development of Blaze aka Bazel, which prides itself on reproducibility.)