8 ms·

I don't get how someone achieves reproducibility of builds: what about files metadata like creation/modification timestamps? Do they forge them? Or are these da

I don't get how someone achieves reproducibility of builds: what about files metadata like creation/modification timestamps? Do they forge them? Or are these data treated as not important enough (like it 2 files with different metadata but identical contents should have the same checksum when hashed)?

HideousKojima 2y ago
Those aren't needed to generate a hash of a file. And that metadata isn't part of the file itself (or at least doesn't need to be), it's part of the filesystem or OS
- imcritic 2y ago
  That's an acceptable answer for the simple case when you distribute just a file, but what if your distribution is something more complex, like an archive with some sub-archives? Metadata in the internal files will affect the checksum of the resulting archive.
  - exe34 2y ago
    unless you fix them to a known epoch.
  - londons_explore 2y ago
    Finding and fixing cases like this are part of what the project has done...
c0l0 2y ago
Yes.
o11c 2y ago
Timestamps are easiest part - you just set everything according to the chosen epoch. The hard things involve things like unstable hash orderings, non-sorted filesystem listing, parallel execution, address-space randomization, ...
- koolba 2y ago
  ASLR shouldn’t be an issue unless you intend to capture the entire memory state of the application. It’s an intermediate representation in memory, not an output of any given step of a build. Annoying edge cases come up for things like internal object serialization to sort things like JSON keys in config files.
  - sodality2 2y ago
    Let’s say a compiler is doing something in a multi-threaded manner - isn’t it possible that ASLR would affect the ordering of certain events which could change the compiled output? Sure you could just set threads to 1 but there’s probably some more edge cases in there I haven’t thought of.
    - zamadatix 2y ago
      I think you'd need the compiler to guarantee serialization order of such operations regardless if you used ASLR or not. Otherwise you're just hoping thread scheduling, core clocking, thread memory access, and many other things are the same between every system trying to do a reproducible build. Even setting threads to 1 may not solve that problem class if asynchronous functions/syscalls come into play.
  - cperciva 2y ago
    FreeBSD tripped over an issue recently where a C++ program (I think clang?) used a collection of pointers and output values in an order based on the pointers rather than the values they pointed to. ASLR by itself shouldn't cause reproducibility issues, but it can certainly expose bugs.
    - ahartmetz 2y ago
      It is sometimes just fine to have a hash table with pointers as keys. It is by design an unordered collection, so you do not care about the order, only about finding entries. Then at some point you happen to need all the entries, you iterate, and you get a random order. Which is not necessarily a problem unless you want reproducible builds, which is just a new requirement, not exposing a latent bug.
purkka 2y ago
Generally, yes: https://reproducible-builds.org/docs/timestamps/ https://reproducible-builds.org/docs/timestamps/ Since the build is reproducible, it should not matter when it was built. If you want to trace a build back to its source, there are much better ways than a timestamp.
- ryandrake 2y ago
  C compilers offer __DATE__ and __TIME__ macros, which expand to string constants that describe the date and time that the preprocessor was invoked. Any code using these would have different strings each time it was built, and would need to be modified. I can't think of a good reason for them to be used in an actual production program, but for whatever reason, they exist.
  - fmbb 2y ago
    Toolchains for reproducible software likely let you set these values, or ensure they are 1970-01-01 00:00:00
    - mikepurvis 2y ago
      Nix sets everything to the epoch, although I believe Debian's approach is to just use the date of the newest file in the dsc tarballs.
      - yjftsjthsd-h 2y ago
        Nix can also set it to things other than 0; I think my favorite is to set it by the time of the commit from which you're building.
        deleted 2y ago
        [deleted]
        terinjokes 2y ago
        Which is also used when the contents of a derivation will be included in a zip file. The Unix epoch is about a decade older than the zip epoch.
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
jzb 2y ago
Debian uses a tool called `strip-nondeterminism` to help with this in part: https://salsa.debian.org/reproducible-builds/strip-nondeterminism https://salsa.debian.org/reproducible-builds/strip-nondeterm... There's lots of info on the Debian site about their reproducibility efforts, and there's a story from 2024's DebConf that may be of interest: https://lwn.net/Articles/985739/ https://lwn.net/Articles/985739/
- frakkingcylons 2y ago
  I see this is written in Perl, is that the case with most Debian tooling?
  - dannyobrien 2y ago
    some, but not all. There's a bunch of historical code which means that Perl is in the base install, but modern tooling has a lot of Python too, as well as POSIX shell (not bash).
    - alfiedotwtf 2y ago
      Though a lot of the apt tooling is definitely written in Perl the last time I had to deep dive
      - johnisgood 2y ago
        And a lot of OpenBSD-related stuff is written in Perl, too. I do not think it is a bad thing at all.
        alfiedotwtf 2y ago
        I absolutely love Perl. I'm just so sad Python won because Google blessed it as a language and at the time everyone wanted to work for Google. Perl always gets hate on HN, but I actually wonder of those commenter, who has actually spent over a single hours using Perl after they've read the Camel book. Honest opinion: if you're going to be spending time in Linux in your career, then you should read the Camel book at least once. Then and only then should you get to have an opinion on Perl!
echoangle 2y ago
Maybe dumb question but why would this change the reproducibility? If you clone a git repo, do you not get the meta data as it is stored in git? Or would the files have the modification date of the cloning? I never actually checked that.
- mathfailure 2y ago
  You clone source from git, but then you use them to build some artifacts. The artifacts build time may differ, yet with reproducible builds - the artifact should match.
  - echoangle 2y ago
    Right, but if you only clone and build, why would the files modification date be different compared to the version that was committed to git? Does just cloning a repo already lead to different file modification dates in my local copy?
    - hoten 2y ago
      Git does not store or restore file modification times.
      - codetrotter 2y ago
        And the reason for that in turn is because if you are on one commit and check out and older commit, then restoring file modification times to what they were at the time of the older commit would cause build tools that look at file modification times to sometimes not pick up on all the changes.
      - echoangle 2y ago
        Ah ok, that explains it.
paulddraper 2y ago
> Do they forge them? Yes. All archive entries and date source code macros and any other timestamps are set to a standardized date (in the past).
- lamby 2y ago
  This is not quite right. At least in Debian, only files that are newer than some standardised date are to that standardised date. This "clamping" preserves any metadata in older files.
TacticalCoder 2y ago
> ... what about files metadata like creation/modification timestamps? Do they forge them? The least difficult to solve for reproducible build but yes. The real question is: why, in the past, was an entire ecosystem created where non-determinism was the norm and everybody thought it was somehow ok? Instead of asking: "how one achieves reproducibility?" we may wonder "why did people got out of their way to make sure something as simple as a timestamp would screw determinism?". For that's the anti-security mindset we have to fight. And Debian did.
- BobbyTables2 2y ago
  You’re forgetting that source control used to not be a mainstream practice… Software was more artisanal in nature…
- brohee 2y ago
  TBH security is someone the source of the issues, as it often involves adding randomness. For example, replacing deterministic hashes by keyed hashes to protect from hash flooding DoS led to deterministic output becoming nondeterministic (e.g. when displaying a hash table in its natural order). Sorting had to be added to that kind of output.

HideousKojima 2y ago

Those aren't needed to generate a hash of a file. And that metadata isn't part of the file itself (or at least doesn't need to be), it's part of the filesystem or OS

imcritic 2y ago
That's an acceptable answer for the simple case when you distribute just a file, but what if your distribution is something more complex, like an archive with some sub-archives? Metadata in the internal files will affect the checksum of the resulting archive.
- exe34 2y ago
  unless you fix them to a known epoch.
- londons_explore 2y ago
  Finding and fixing cases like this are part of what the project has done...

c0l0 2y ago

Yes.

o11c 2y ago

Timestamps are easiest part - you just set everything according to the chosen epoch. The hard things involve things like unstable hash orderings, non-sorted filesystem listing, parallel execution, address-space randomization, ...

koolba 2y ago
ASLR shouldn’t be an issue unless you intend to capture the entire memory state of the application. It’s an intermediate representation in memory, not an output of any given step of a build. Annoying edge cases come up for things like internal object serialization to sort things like JSON keys in config files.
- sodality2 2y ago
  Let’s say a compiler is doing something in a multi-threaded manner - isn’t it possible that ASLR would affect the ordering of certain events which could change the compiled output? Sure you could just set threads to 1 but there’s probably some more edge cases in there I haven’t thought of.
  - zamadatix 2y ago
    I think you'd need the compiler to guarantee serialization order of such operations regardless if you used ASLR or not. Otherwise you're just hoping thread scheduling, core clocking, thread memory access, and many other things are the same between every system trying to do a reproducible build. Even setting threads to 1 may not solve that problem class if asynchronous functions/syscalls come into play.
- cperciva 2y ago
  FreeBSD tripped over an issue recently where a C++ program (I think clang?) used a collection of pointers and output values in an order based on the pointers rather than the values they pointed to. ASLR by itself shouldn't cause reproducibility issues, but it can certainly expose bugs.
  - ahartmetz 2y ago
    It is sometimes just fine to have a hash table with pointers as keys. It is by design an unordered collection, so you do not care about the order, only about finding entries. Then at some point you happen to need all the entries, you iterate, and you get a random order. Which is not necessarily a problem unless you want reproducible builds, which is just a new requirement, not exposing a latent bug.

purkka 2y ago

Generally, yes: https://reproducible-builds.org/docs/timestamps/ https://reproducible-builds.org/docs/timestamps/ Since the build is reproducible, it should not matter when it was built. If you want to trace a build back to its source, there are much better ways than a timestamp.

ryandrake 2y ago
C compilers offer __DATE__ and __TIME__ macros, which expand to string constants that describe the date and time that the preprocessor was invoked. Any code using these would have different strings each time it was built, and would need to be modified. I can't think of a good reason for them to be used in an actual production program, but for whatever reason, they exist.
- fmbb 2y ago
  Toolchains for reproducible software likely let you set these values, or ensure they are 1970-01-01 00:00:00
  - mikepurvis 2y ago
    Nix sets everything to the epoch, although I believe Debian's approach is to just use the date of the newest file in the dsc tarballs.
    - yjftsjthsd-h 2y ago
      Nix can also set it to things other than 0; I think my favorite is to set it by the time of the commit from which you're building.
      - deleted 2y ago
        [deleted]
      - terinjokes 2y ago
        Which is also used when the contents of a derivation will be included in a zip file. The Unix epoch is about a decade older than the zip epoch.

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]

jzb 2y ago

Debian uses a tool called `strip-nondeterminism` to help with this in part: https://salsa.debian.org/reproducible-builds/strip-nondeterminism https://salsa.debian.org/reproducible-builds/strip-nondeterm... There's lots of info on the Debian site about their reproducibility efforts, and there's a story from 2024's DebConf that may be of interest: https://lwn.net/Articles/985739/ https://lwn.net/Articles/985739/

frakkingcylons 2y ago
I see this is written in Perl, is that the case with most Debian tooling?
- dannyobrien 2y ago
  some, but not all. There's a bunch of historical code which means that Perl is in the base install, but modern tooling has a lot of Python too, as well as POSIX shell (not bash).
  - alfiedotwtf 2y ago
    Though a lot of the apt tooling is definitely written in Perl the last time I had to deep dive
    - johnisgood 2y ago
      And a lot of OpenBSD-related stuff is written in Perl, too. I do not think it is a bad thing at all.
      - alfiedotwtf 2y ago
        I absolutely love Perl. I'm just so sad Python won because Google blessed it as a language and at the time everyone wanted to work for Google. Perl always gets hate on HN, but I actually wonder of those commenter, who has actually spent over a single hours using Perl after they've read the Camel book. Honest opinion: if you're going to be spending time in Linux in your career, then you should read the Camel book at least once. Then and only then should you get to have an opinion on Perl!

echoangle 2y ago

Maybe dumb question but why would this change the reproducibility? If you clone a git repo, do you not get the meta data as it is stored in git? Or would the files have the modification date of the cloning? I never actually checked that.

mathfailure 2y ago
You clone source from git, but then you use them to build some artifacts. The artifacts build time may differ, yet with reproducible builds - the artifact should match.
- echoangle 2y ago
  Right, but if you only clone and build, why would the files modification date be different compared to the version that was committed to git? Does just cloning a repo already lead to different file modification dates in my local copy?
  - hoten 2y ago
    Git does not store or restore file modification times.
    - codetrotter 2y ago
      And the reason for that in turn is because if you are on one commit and check out and older commit, then restoring file modification times to what they were at the time of the older commit would cause build tools that look at file modification times to sometimes not pick up on all the changes.
    - echoangle 2y ago
      Ah ok, that explains it.

paulddraper 2y ago

> Do they forge them? Yes. All archive entries and date source code macros and any other timestamps are set to a standardized date (in the past).

lamby 2y ago
This is not quite right. At least in Debian, only files that are newer than some standardised date are to that standardised date. This "clamping" preserves any metadata in older files.

TacticalCoder 2y ago

> ... what about files metadata like creation/modification timestamps? Do they forge them? The least difficult to solve for reproducible build but yes. The real question is: why, in the past, was an entire ecosystem created where non-determinism was the norm and everybody thought it was somehow ok? Instead of asking: "how one achieves reproducibility?" we may wonder "why did people got out of their way to make sure something as simple as a timestamp would screw determinism?". For that's the anti-security mindset we have to fight. And Debian did.

BobbyTables2 2y ago
You’re forgetting that source control used to not be a mainstream practice… Software was more artisanal in nature…
brohee 2y ago
TBH security is someone the source of the issues, as it often involves adding randomness. For example, replacing deterministic hashes by keyed hashes to protect from hash flooding DoS led to deterministic output becoming nondeterministic (e.g. when displaying a hash table in its natural order). Sorting had to be added to that kind of output.