3 ms·

Depending on the tools used, it can be "easy" for things in the environment to leak into the build. Timestamps are a common bane of reproducible builds. I under

Depending on the tools used, it can be "easy" for things in the environment to leak into the build. Timestamps are a common bane of reproducible builds. I understand that a non-trivial amount of reproducible build work consists of hunting down every timestamp in a build, and then finding a way to inject alternate timestamps into them. For example if we just look at one of the listed reproducibility bugs in the report (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864082 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864082) you can get a flavor of the random bits and bods that need to be hunted down. We could say that a system can be relatively easily made reproducible if starting from scratch - certainly all of the tooling is there. The far more significant challenge is retrofitting reproducibility into existing large projects.

k__ 5y ago
Why are people using timestamps in builds and what is the better alternative?
- capableweb 5y ago
  Some people want to know when something was built. Better option is to use Git commit of build or even last-modified of the file it's being embedded into. Alternatively, allow a env var to override what date/time the software thinks it is.
- richbell 5y ago
  > Why are people using timestamps in builds and what is the better alternative? Timestamps are often an implicit part of the build which leads to the same code producing 'different' artifacts (e.g. file created timestamp, generated metadata about the build stored in the artifact itself). One alternative is to support explicitly defining the 'build timestamp' so all builds with the same timestamp produces the same artifact. This is what Maven does[0]; I'd imagine it's much harder for build systems that download and recompile dependencies. [0] https://maven.apache.org/guides/mini/guide-reproducible-builds.html https://maven.apache.org/guides/mini/guide-reproducible-buil...
- michaelt 5y ago
  First of all, every filesystem has timestamps and they've got to be set to some value. Second of all, it's useful to have some sort of version marker that lets you tell at a glance if two copies of the software are the same, or what code was running when that log file was created. Build timestamps are a classic way of doing this in a version-control-system-independent way, without advancing the 'headline' version number for every developer's test builds. And of course, if you include things like debug symbols it's useful to be able to tell if the source files on disk have changed since the build took place, as the line numbers could be all off. Alternatives include using a git commit hash; having a separate build pipeline for public release versions then not worrying if developers' builds aren't reproducible; and just being careful with developers' test builds.
  - 400thecat 5y ago
    but why should different timestamps on the filesystem cause different binary?
    - icegreentea2 5y ago
      Don't think binaries, think build outputs. Imagine part of your build is generating a bunch of different files, and then packaging into an archive file, or into your installer bundle (or whatever). If your archive preserves file timestamps, then your end up with different archive files.
    - comex 5y ago
      There’s a long list of tools that follow a general pattern: they take an input file and produce an output file, but the output file doesn’t just depend on the contents of the input file; it also embeds its modification time somewhere. Sometimes the goal is to be able to automatically regenerate the output if the input changes, by comparing timestamps. Sometimes it’s just intended as general-purpose metadata. Debian’s reproducible builds wiki page has many examples: https://wiki.debian.org/ReproducibleBuilds/Howto#Files_in_data.tar_contain_timestamps https://wiki.debian.org/ReproducibleBuilds/Howto#Files_in_da... Notably, this affects archivers such as tar and zip, but there’s also a long tail of application-specific tools.

k__ 5y ago

Why are people using timestamps in builds and what is the better alternative?

capableweb 5y ago
Some people want to know when something was built. Better option is to use Git commit of build or even last-modified of the file it's being embedded into. Alternatively, allow a env var to override what date/time the software thinks it is.
richbell 5y ago
> Why are people using timestamps in builds and what is the better alternative? Timestamps are often an implicit part of the build which leads to the same code producing 'different' artifacts (e.g. file created timestamp, generated metadata about the build stored in the artifact itself). One alternative is to support explicitly defining the 'build timestamp' so all builds with the same timestamp produces the same artifact. This is what Maven does[0]; I'd imagine it's much harder for build systems that download and recompile dependencies. [0] https://maven.apache.org/guides/mini/guide-reproducible-builds.html https://maven.apache.org/guides/mini/guide-reproducible-buil...
michaelt 5y ago
First of all, every filesystem has timestamps and they've got to be set to some value. Second of all, it's useful to have some sort of version marker that lets you tell at a glance if two copies of the software are the same, or what code was running when that log file was created. Build timestamps are a classic way of doing this in a version-control-system-independent way, without advancing the 'headline' version number for every developer's test builds. And of course, if you include things like debug symbols it's useful to be able to tell if the source files on disk have changed since the build took place, as the line numbers could be all off. Alternatives include using a git commit hash; having a separate build pipeline for public release versions then not worrying if developers' builds aren't reproducible; and just being careful with developers' test builds.
- 400thecat 5y ago
  but why should different timestamps on the filesystem cause different binary?
  - icegreentea2 5y ago
    Don't think binaries, think build outputs. Imagine part of your build is generating a bunch of different files, and then packaging into an archive file, or into your installer bundle (or whatever). If your archive preserves file timestamps, then your end up with different archive files.
  - comex 5y ago
    There’s a long list of tools that follow a general pattern: they take an input file and produce an output file, but the output file doesn’t just depend on the contents of the input file; it also embeds its modification time somewhere. Sometimes the goal is to be able to automatically regenerate the output if the input changes, by comparing timestamps. Sometimes it’s just intended as general-purpose metadata. Debian’s reproducible builds wiki page has many examples: https://wiki.debian.org/ReproducibleBuilds/Howto#Files_in_data.tar_contain_timestamps https://wiki.debian.org/ReproducibleBuilds/Howto#Files_in_da... Notably, this affects archivers such as tar and zip, but there’s also a long tail of application-specific tools.