7 ms·

Debian reproducibility statistics

188 points by lamby 9y ago

pavlov 9y ago
The lack of a .yet TLD is a real missed opportunity.
- penpapersw 9y ago
  Meh, joke sites like this aren't nearly as prominent as sites for apps, I'd really like a .app TLD instead. That said, the distinction between app and service is blurring a lot, so Spotify and VS Code probably both qualify for .app but one also has a web interface. Everything is confusing, let's just stick to .com
  - thinkMOAR 9y ago
    Why do you consider this a joke site? And just because you are easily confused, everybody should stick to .com only?
    - forbiddenlake 9y ago
      The link was originally to http://isdebianreproducibleyet.com/ http://isdebianreproducibleyet.com/ .
bialpio 9y ago
Can someone shed some light on what exactly "reproducible" means in this context?
- dozzie 9y ago
  It's a term for binaries (usually ELF) being byte-to-byte equal in two different runs of a compiler. This way you can build a binary package from source package and if its content is the same, you know what source code was used to build the package, and then you can e.g. inspect the code for backdoors or build debug symbols without planning for that beforehand. https://en.wikipedia.org/wiki/Reproducible_build https://en.wikipedia.org/wiki/Reproducible_build
  - gnasr 9y ago
    And why Debian is not reproducible?
    - wiz21c 9y ago
      because during the compilation, some package use date-dependent stuff, for example (I dunno the exact details though)
      - vertex-four 9y ago
        Also some build systems create artifacts as a result of timing-dependent algorithms. Simply put, if two things A and B run simultaneously, and A completes before B, then in some compilers/build systems/etc the result can be different from B completing before A. GHC, as a well-known example, suffers from this problem.
    - Daviey 9y ago
      Often, package is built using a certain library version (from a different package), that library is then updated - and the new package cannot build using the new library. Or... deeper parts of the compiler toolchain change, and the application doesn't re-build without changes.
      - vertex-four 9y ago
        That has nothing to do with reproducible builds - the issue being solved by reproducible builds is that even in exactly the same environment, running the same build script twice can result in differences in the output.
        noselasd 9y ago
        Surely it can't have "nothing to do" ? Many debian source packages specifies dependencies using >= , something has to account for performing a build using the same minor version of such a dependency.
- penpapersw 9y ago
  More details at https://reproducible-builds.org/ https://reproducible-builds.org/ which is linked to at the bottom of OP's page.
agumonkey 9y ago
What a nice progression https://tests.reproducible-builds.org/debian/stats_issues.png https://tests.reproducible-builds.org/debian/stats_issues.pn... Project started in 2014 it seems.
- Mchl 9y ago
  I'd invest in that
  - tome 9y ago
    But would you have been able to hold your nerve through May?
hmmm___food 9y ago
Maybe im missing something but it seems the justification behind this is based on a situation in which source wasnt open. Debian is open so why is reproducibility a priority?
- kalmi10 9y ago
  So that anyone can check whether a binary (on the mirrors) has been tampered with in ways that are not present in the source.
- yakubin 9y ago
  On the contrary. Reproducibility means that given the source you are able to produce the exact same binary, i.e. you can verify that someone didn't modify it before building. When software isn't open, then reproducibility is meaningless, because you don't have the source, so: 1. you can't verify anything (you can't compile anything); 2. assurance of the binary not being modified gives you nothing, since you don't know if there is no malicious code in the original source code.
- thechao 9y ago
  I know Debian has lofty goals with respect to reproducibility, but on a purely "I hate waiting for builds", getting deterministic object files can prevent spurious linking. For example: change a comment, watch your code link for 10 minutes.
  - lamby 9y ago
    Mmm. Think of the CO2/power savings at a big coding shop like, I dunno, Google...
- wiz21c 9y ago
  Just to give an example, think about some software you put in a voting machine. Even if the code is open source and given to voters, they need to be able to compile it themselves and compare their compiled version to the one in the voting machine... (well, to be honest, this only makes sense if the voting machine runs the compiled code only and doesn't sneak other stuff in at runtime, but that's another story)
vegbrasil 9y ago
Is Docker (or any other container platform) a facilitator to reproducible builds? Making the environment standard between builds is probably easier in a container.
- georgyo 9y ago
  Builds have been happening in single use chroots for a very long time. The problem is if code includes the current date, or generates some randomness at build time. These issues must be identified and patched.
- 0xcde4c3db 9y ago
  Docker is part of a broader "reproducible build environment" strategy, but doesn't really help with some of the things that cause problems (timestamps, kernel version, random IDs).
  - wtallis 9y ago
    Docker seeks to reproduce a functionally equivalent software environment, motivated by version management concerns. Debian is trying to reproduce bitwise identical build products, motivated by security concerns.
- samueloph 9y ago
  Debian builders already run on sbuild[1], which uses chroot, so it's already a contained environment. I'm not sure how the reproducible builds machines works for forcing two different build results, but i'd bet they're using sbuild too. Here's a diffoscope of one of my packages which is not reproducible right now[2]. I also highly recommend the reproducible builds talk at this year's debconf with the repro build folks[3]. [1]https://wiki.debian.org/sbuild https://wiki.debian.org/sbuild [2]https://tests.reproducible-builds.org/debian/rb-pkg/unstable/i386/diffoscope-results/aircrack-ng.html https://tests.reproducible-builds.org/debian/rb-pkg/unstable... [3]https://debconf17.debconf.org/talks/14/ https://debconf17.debconf.org/talks/14/
- zdw 9y ago
  Docker images built with the first-party toolchain aren't reproducible - if you run `docker build ...` on a Dockerfile, then delete the image and rerun it, you'll get a different set of image hashes. This is likely due to timestamp embedding. There are other toolsets that supposedly create byte-identical Docker images generation (Bazel, some others), but I haven't tried them.
lamby 9y ago
Mods, how come my post was completely edited? :)
- rahiel 9y ago
  This was the originally submitted link: http://isdebianreproducibleyet.com/ http://isdebianreproducibleyet.com/
  - lamby 9y ago
    Indeed, which is a completely different "view" on things...
    - sctb 9y ago
      A moderator edited the URL from a page that says "NO" which links to this.
      - lamby 9y ago
        I'm aware of that. I was asking why that change was made.
        yorwba 9y ago
        From the guidelines: Please submit the original source. If a post reports on something found on another site, submit the latter.
        deleted 9y ago
        [deleted]
      - heavenlyhash 9y ago
        It is amusing (well, to me) to note that ^lamby here is Chris Lamb, a major contributor to the Debian Reproducible Builds project. If anyone has the social currency to publish a such a capslock "no" and still mean it in a purely informative and constructive way, he probably is towards the top of the list of folks so qualified.
entelechy 9y ago
I wonder if switching from cmake to buck[1] and buckaroo[2] would simplify and increase reproducibility. [1] https://buckbuild.com https://buckbuild.com [2] https://buckaroo.pm https://buckaroo.pm
- philippnagel 9y ago
  Also interesting: https://bazel.build/ https://bazel.build/
  - yorwba 9y ago
    Funny that you should mention bazel on the topic of reproducible builds. When I last used it to build TensorFlow, the build was very much not reproducible. I took me three rebuilds with same flags to finally get GPU support compiled in.
- moosingin3space 9y ago
  Based on the experiences of systemd, gtk, and X11 in switching to Meson, I'd think Meson might be the best choice here. While Buck, Bazel, Pants, etc. are designed for large projects, Meson is designed for small-to-medium-size projects and integrates with pkg-config, which in turn should provide simpler integration with distribution package managers. My experiences with Bazel demonstrated that integration with distribution packages can be quite difficult.
  - entelechy 9y ago
    Although never mentioned, you can use pkg-config with buck by embedding your pkg-config command into the compiler flags eg.: cxxflags = ${pkg-config opencv python --cflags --libs} Furthermore I don't see the reason why you woudn't use buck for small projects. What makes you think buck is less suitable for small projects ?
  - entelechy 9y ago
    why I'm consider that using pkg-config in buck may break reproducibility: https://github.com/facebook/buck/issues/1430 https://github.com/facebook/buck/issues/1430
YouKnowBetter 9y ago
On a side note: A point that many a company ignores when they demand some assurance so their software vendor a) can be held liable b) will hand over the source in case of drama, is to give the source code in Escrow. The number one error these companies make is to think that source code without a reproducible build environment means anything at all. Reproducibility is not easy. I applaud Debian (and in the older days TrueCrypt) for giving this more exposure.
- bluGill 9y ago
  Someplace my company has a closet with a computer containing Windows XP (not sure which service pack, but not the latest), and and old version of visual studio, just in case we need to fix a bug. I know we still have it because the project got resurrected recently and we had to clone the harddrive a few times because nobody could find a copy of that version of visual studio that can be installed. (Microsoft dropped support for whatever version of WinCE with no upgrade path)
  - monocasa 9y ago
    We have a bunch of VMs cloned from ancient machines for the same reason.
mmagin 9y ago
See also this page of tools: https://reproducible-builds.org/tools/ https://reproducible-builds.org/tools/ disorderfs is nice for ferreting out readdir()-ordering effects.

188 points by lamby 9y ago

pavlov 9y ago

The lack of a .yet TLD is a real missed opportunity.

penpapersw 9y ago
Meh, joke sites like this aren't nearly as prominent as sites for apps, I'd really like a .app TLD instead. That said, the distinction between app and service is blurring a lot, so Spotify and VS Code probably both qualify for .app but one also has a web interface. Everything is confusing, let's just stick to .com
- thinkMOAR 9y ago
  Why do you consider this a joke site? And just because you are easily confused, everybody should stick to .com only?
  - forbiddenlake 9y ago
    The link was originally to http://isdebianreproducibleyet.com/ http://isdebianreproducibleyet.com/ .

bialpio 9y ago

Can someone shed some light on what exactly "reproducible" means in this context?

dozzie 9y ago
It's a term for binaries (usually ELF) being byte-to-byte equal in two different runs of a compiler. This way you can build a binary package from source package and if its content is the same, you know what source code was used to build the package, and then you can e.g. inspect the code for backdoors or build debug symbols without planning for that beforehand. https://en.wikipedia.org/wiki/Reproducible_build https://en.wikipedia.org/wiki/Reproducible_build
- gnasr 9y ago
  And why Debian is not reproducible?
  - wiz21c 9y ago
    because during the compilation, some package use date-dependent stuff, for example (I dunno the exact details though)
    - vertex-four 9y ago
      Also some build systems create artifacts as a result of timing-dependent algorithms. Simply put, if two things A and B run simultaneously, and A completes before B, then in some compilers/build systems/etc the result can be different from B completing before A. GHC, as a well-known example, suffers from this problem.
  - Daviey 9y ago
    Often, package is built using a certain library version (from a different package), that library is then updated - and the new package cannot build using the new library. Or... deeper parts of the compiler toolchain change, and the application doesn't re-build without changes.
    - vertex-four 9y ago
      That has nothing to do with reproducible builds - the issue being solved by reproducible builds is that even in exactly the same environment, running the same build script twice can result in differences in the output.
      - noselasd 9y ago
        Surely it can't have "nothing to do" ? Many debian source packages specifies dependencies using >= , something has to account for performing a build using the same minor version of such a dependency.
penpapersw 9y ago
More details at https://reproducible-builds.org/ https://reproducible-builds.org/ which is linked to at the bottom of OP's page.

agumonkey 9y ago

What a nice progression https://tests.reproducible-builds.org/debian/stats_issues.png https://tests.reproducible-builds.org/debian/stats_issues.pn... Project started in 2014 it seems.

Mchl 9y ago
I'd invest in that
- tome 9y ago
  But would you have been able to hold your nerve through May?

hmmm___food 9y ago

Maybe im missing something but it seems the justification behind this is based on a situation in which source wasnt open. Debian is open so why is reproducibility a priority?

kalmi10 9y ago
So that anyone can check whether a binary (on the mirrors) has been tampered with in ways that are not present in the source.
yakubin 9y ago
On the contrary. Reproducibility means that given the source you are able to produce the exact same binary, i.e. you can verify that someone didn't modify it before building. When software isn't open, then reproducibility is meaningless, because you don't have the source, so: 1. you can't verify anything (you can't compile anything); 2. assurance of the binary not being modified gives you nothing, since you don't know if there is no malicious code in the original source code.
thechao 9y ago
I know Debian has lofty goals with respect to reproducibility, but on a purely "I hate waiting for builds", getting deterministic object files can prevent spurious linking. For example: change a comment, watch your code link for 10 minutes.
- lamby 9y ago
  Mmm. Think of the CO2/power savings at a big coding shop like, I dunno, Google...
wiz21c 9y ago
Just to give an example, think about some software you put in a voting machine. Even if the code is open source and given to voters, they need to be able to compile it themselves and compare their compiled version to the one in the voting machine... (well, to be honest, this only makes sense if the voting machine runs the compiled code only and doesn't sneak other stuff in at runtime, but that's another story)

vegbrasil 9y ago

Is Docker (or any other container platform) a facilitator to reproducible builds? Making the environment standard between builds is probably easier in a container.

georgyo 9y ago
Builds have been happening in single use chroots for a very long time. The problem is if code includes the current date, or generates some randomness at build time. These issues must be identified and patched.
0xcde4c3db 9y ago
Docker is part of a broader "reproducible build environment" strategy, but doesn't really help with some of the things that cause problems (timestamps, kernel version, random IDs).
- wtallis 9y ago
  Docker seeks to reproduce a functionally equivalent software environment, motivated by version management concerns. Debian is trying to reproduce bitwise identical build products, motivated by security concerns.
samueloph 9y ago
Debian builders already run on sbuild[1], which uses chroot, so it's already a contained environment. I'm not sure how the reproducible builds machines works for forcing two different build results, but i'd bet they're using sbuild too. Here's a diffoscope of one of my packages which is not reproducible right now[2]. I also highly recommend the reproducible builds talk at this year's debconf with the repro build folks[3]. [1]https://wiki.debian.org/sbuild https://wiki.debian.org/sbuild [2]https://tests.reproducible-builds.org/debian/rb-pkg/unstable/i386/diffoscope-results/aircrack-ng.html https://tests.reproducible-builds.org/debian/rb-pkg/unstable... [3]https://debconf17.debconf.org/talks/14/ https://debconf17.debconf.org/talks/14/
zdw 9y ago
Docker images built with the first-party toolchain aren't reproducible - if you run `docker build ...` on a Dockerfile, then delete the image and rerun it, you'll get a different set of image hashes. This is likely due to timestamp embedding. There are other toolsets that supposedly create byte-identical Docker images generation (Bazel, some others), but I haven't tried them.

lamby 9y ago

Mods, how come my post was completely edited? :)

rahiel 9y ago
This was the originally submitted link: http://isdebianreproducibleyet.com/ http://isdebianreproducibleyet.com/
- lamby 9y ago
  Indeed, which is a completely different "view" on things...
  - sctb 9y ago
    A moderator edited the URL from a page that says "NO" which links to this.
    - lamby 9y ago
      I'm aware of that. I was asking why that change was made.
      - yorwba 9y ago
        From the guidelines: Please submit the original source. If a post reports on something found on another site, submit the latter.
      - deleted 9y ago
        [deleted]
    - heavenlyhash 9y ago
      It is amusing (well, to me) to note that ^lamby here is Chris Lamb, a major contributor to the Debian Reproducible Builds project. If anyone has the social currency to publish a such a capslock "no" and still mean it in a purely informative and constructive way, he probably is towards the top of the list of folks so qualified.

entelechy 9y ago

I wonder if switching from cmake to buck[1] and buckaroo[2] would simplify and increase reproducibility. [1] https://buckbuild.com https://buckbuild.com [2] https://buckaroo.pm https://buckaroo.pm

philippnagel 9y ago
Also interesting: https://bazel.build/ https://bazel.build/
- yorwba 9y ago
  Funny that you should mention bazel on the topic of reproducible builds. When I last used it to build TensorFlow, the build was very much not reproducible. I took me three rebuilds with same flags to finally get GPU support compiled in.
moosingin3space 9y ago
Based on the experiences of systemd, gtk, and X11 in switching to Meson, I'd think Meson might be the best choice here. While Buck, Bazel, Pants, etc. are designed for large projects, Meson is designed for small-to-medium-size projects and integrates with pkg-config, which in turn should provide simpler integration with distribution package managers. My experiences with Bazel demonstrated that integration with distribution packages can be quite difficult.
- entelechy 9y ago
  Although never mentioned, you can use pkg-config with buck by embedding your pkg-config command into the compiler flags eg.: cxxflags = ${pkg-config opencv python --cflags --libs} Furthermore I don't see the reason why you woudn't use buck for small projects. What makes you think buck is less suitable for small projects ?
- entelechy 9y ago
  why I'm consider that using pkg-config in buck may break reproducibility: https://github.com/facebook/buck/issues/1430 https://github.com/facebook/buck/issues/1430

YouKnowBetter 9y ago

On a side note: A point that many a company ignores when they demand some assurance so their software vendor a) can be held liable b) will hand over the source in case of drama, is to give the source code in Escrow. The number one error these companies make is to think that source code without a reproducible build environment means anything at all. Reproducibility is not easy. I applaud Debian (and in the older days TrueCrypt) for giving this more exposure.

bluGill 9y ago
Someplace my company has a closet with a computer containing Windows XP (not sure which service pack, but not the latest), and and old version of visual studio, just in case we need to fix a bug. I know we still have it because the project got resurrected recently and we had to clone the harddrive a few times because nobody could find a copy of that version of visual studio that can be installed. (Microsoft dropped support for whatever version of WinCE with no upgrade path)
- monocasa 9y ago
  We have a bunch of VMs cloned from ancient machines for the same reason.

mmagin 9y ago

See also this page of tools: https://reproducible-builds.org/tools/ https://reproducible-builds.org/tools/ disorderfs is nice for ferreting out readdir()-ordering effects.