43 ms·

One has to step back and ask: "What made you decide to introduce non-determinism into your compilation process in the first place?"

dzdt 8y ago
If you aren't thinking about reproducibility things like putting timestamps in the binary seem reasonable. Which is what lots of tools did. Hence the problem.
jamieson-becker 8y ago
You don't really 'introduce non-determinism'. That's like saying, "why did you introduce entropy?" It's there. It happens, and it's difficult to get rid of in any complex system. https://en.wikipedia.org/wiki/Reproducible_builds https://en.wikipedia.org/wiki/Reproducible_builds https://wiki.debian.org/ReproducibleBuilds https://wiki.debian.org/ReproducibleBuilds https://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html https://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html Debian was one of the first large distributions to use a build farm for many different architectures. Maintaining deterministic builds (even as far as just file timestamps) in a distributed environment is very challenging.
pm215 8y ago
Nobody sat down and said "yes, I'll make this nondeterministic". If you look at the wiki page's sampling of different issues -- https://wiki.debian.org/ReproducibleBuilds/Howto#Identified_problems.2C_and_possible_solutions https://wiki.debian.org/ReproducibleBuilds/Howto#Identified_... -- you'll see it's a mix of various things: * output is deterministic but dependent on some aspect of the build environment (locale, hostname, etc) * output is accidentally non-deterministic (eg "we put all the .html files into a tarball with a shell glob pattern, which gets you an order dependent on your filesystem implementation and the phase of the moon") * a wide array of "output contains a timestamp" issues "Let's put the timestamp into our version string/a generated file so the user knows when it was built" is a really common thing, and it seemed like purely a nice convenience feature until the concept of 100% binary-reproducible builds became a current concern.
- jlarocco 8y ago
  Except for the first one, the last three are great examples of why I don't care about 100% binary reproducible builds as an end user. Knowing when something was built is more valuable to me than a reproducible build. Seems like a fad triumphing over common sense.
  - peterwwillis 8y ago
    Users often don't care about things until they become a problem. For example, most users don't care how their password is stored on a server... until it gets compromised. Also, timestamps in output usually indicate source code age, not build date. I have a package with '2015' in the version but it was built in 2018.
    - jlarocco 8y ago
      That's a condescending non-answer. We're not talking about passwords in a database, we're talking about reproducible builds. I already trust Debian, so it's irrelevant to me, as a user, if their builds are 100% reproducible. Can you explain why I should feel differently? Although I don't think it's a valuable thing to spend time on, it's not my time, so I honestly don't care too much.
      - deleted 8y ago
        [deleted]
      - blihp 8y ago
        From an end user point of view, probably the most important thing it allows Debian developers to do is verify that the binary packages they are shipping are the packages they intended to ship. You trust them, they have to trust their tools... so they're making sure their tools tell them about yet another situation when something might have gone wrong. (it could be something innocent or malicious, but without the tooling, they might not even be aware of the issue)
        8y ago
scarejunba 8y ago
This is true and it's also the model I use to avoid bugs. Any time I'd add a bug to the codebase, I just look at it and decide not to write the bug. In this way, no bugs are ever introduced.

dzdt 8y ago

If you aren't thinking about reproducibility things like putting timestamps in the binary seem reasonable. Which is what lots of tools did. Hence the problem.

jamieson-becker 8y ago

You don't really 'introduce non-determinism'. That's like saying, "why did you introduce entropy?" It's there. It happens, and it's difficult to get rid of in any complex system. https://en.wikipedia.org/wiki/Reproducible_builds https://en.wikipedia.org/wiki/Reproducible_builds https://wiki.debian.org/ReproducibleBuilds https://wiki.debian.org/ReproducibleBuilds https://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html https://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html Debian was one of the first large distributions to use a build farm for many different architectures. Maintaining deterministic builds (even as far as just file timestamps) in a distributed environment is very challenging.

pm215 8y ago

Nobody sat down and said "yes, I'll make this nondeterministic". If you look at the wiki page's sampling of different issues -- https://wiki.debian.org/ReproducibleBuilds/Howto#Identified_problems.2C_and_possible_solutions https://wiki.debian.org/ReproducibleBuilds/Howto#Identified_... -- you'll see it's a mix of various things: * output is deterministic but dependent on some aspect of the build environment (locale, hostname, etc) * output is accidentally non-deterministic (eg "we put all the .html files into a tarball with a shell glob pattern, which gets you an order dependent on your filesystem implementation and the phase of the moon") * a wide array of "output contains a timestamp" issues "Let's put the timestamp into our version string/a generated file so the user knows when it was built" is a really common thing, and it seemed like purely a nice convenience feature until the concept of 100% binary-reproducible builds became a current concern.

jlarocco 8y ago
Except for the first one, the last three are great examples of why I don't care about 100% binary reproducible builds as an end user. Knowing when something was built is more valuable to me than a reproducible build. Seems like a fad triumphing over common sense.
- peterwwillis 8y ago
  Users often don't care about things until they become a problem. For example, most users don't care how their password is stored on a server... until it gets compromised. Also, timestamps in output usually indicate source code age, not build date. I have a package with '2015' in the version but it was built in 2018.
  - jlarocco 8y ago
    That's a condescending non-answer. We're not talking about passwords in a database, we're talking about reproducible builds. I already trust Debian, so it's irrelevant to me, as a user, if their builds are 100% reproducible. Can you explain why I should feel differently? Although I don't think it's a valuable thing to spend time on, it's not my time, so I honestly don't care too much.
    - deleted 8y ago
      [deleted]
    - blihp 8y ago
      From an end user point of view, probably the most important thing it allows Debian developers to do is verify that the binary packages they are shipping are the packages they intended to ship. You trust them, they have to trust their tools... so they're making sure their tools tell them about yet another situation when something might have gone wrong. (it could be something innocent or malicious, but without the tooling, they might not even be aware of the issue)
      - 8y ago

scarejunba 8y ago

This is true and it's also the model I use to avoid bugs. Any time I'd add a bug to the codebase, I just look at it and decide not to write the bug. In this way, no bugs are ever introduced.