14 ms·

Debian Buster will only be 54% reproducible, while we could be at 90%

280 points by JNRowe 8y ago

JNRowe 8y ago
Slightly fluid numbers aside, half the packages for Buster still sounds like good progress to me.
- e1ven 8y ago
  It's certainly the right direction! I applaud them for pushing this. From the article, it looks like many of the others are reproducible on a code level, but the release system is using older binaries, which haven't been rebuilt yet. It's too bad it won't be fixed in time for this release, but bodes very well for future releases.
aruggirello 8y ago
BTW I was just looking at diffoscope - developed as part of the "reproducible builds" Debian project. https://try.diffoscope.org/ https://try.diffoscope.org/
peterkelly 8y ago
One has to step back and ask: "What made you decide to introduce non-determinism into your compilation process in the first place?"
- dzdt 8y ago
  If you aren't thinking about reproducibility things like putting timestamps in the binary seem reasonable. Which is what lots of tools did. Hence the problem.
- jamieson-becker 8y ago
  You don't really 'introduce non-determinism'. That's like saying, "why did you introduce entropy?" It's there. It happens, and it's difficult to get rid of in any complex system. https://en.wikipedia.org/wiki/Reproducible_builds https://en.wikipedia.org/wiki/Reproducible_builds https://wiki.debian.org/ReproducibleBuilds https://wiki.debian.org/ReproducibleBuilds https://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html https://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html Debian was one of the first large distributions to use a build farm for many different architectures. Maintaining deterministic builds (even as far as just file timestamps) in a distributed environment is very challenging.
- pm215 8y ago
  Nobody sat down and said "yes, I'll make this nondeterministic". If you look at the wiki page's sampling of different issues -- https://wiki.debian.org/ReproducibleBuilds/Howto#Identified_problems.2C_and_possible_solutions https://wiki.debian.org/ReproducibleBuilds/Howto#Identified_... -- you'll see it's a mix of various things: * output is deterministic but dependent on some aspect of the build environment (locale, hostname, etc) * output is accidentally non-deterministic (eg "we put all the .html files into a tarball with a shell glob pattern, which gets you an order dependent on your filesystem implementation and the phase of the moon") * a wide array of "output contains a timestamp" issues "Let's put the timestamp into our version string/a generated file so the user knows when it was built" is a really common thing, and it seemed like purely a nice convenience feature until the concept of 100% binary-reproducible builds became a current concern.
  - jlarocco 8y ago
    Except for the first one, the last three are great examples of why I don't care about 100% binary reproducible builds as an end user. Knowing when something was built is more valuable to me than a reproducible build. Seems like a fad triumphing over common sense.
    - peterwwillis 8y ago
      Users often don't care about things until they become a problem. For example, most users don't care how their password is stored on a server... until it gets compromised. Also, timestamps in output usually indicate source code age, not build date. I have a package with '2015' in the version but it was built in 2018.
      - jlarocco 8y ago
        That's a condescending non-answer. We're not talking about passwords in a database, we're talking about reproducible builds. I already trust Debian, so it's irrelevant to me, as a user, if their builds are 100% reproducible. Can you explain why I should feel differently? Although I don't think it's a valuable thing to spend time on, it's not my time, so I honestly don't care too much.
- scarejunba 8y ago
  This is true and it's also the model I use to avoid bugs. Any time I'd add a bug to the codebase, I just look at it and decide not to write the bug. In this way, no bugs are ever introduced.
badfrog 8y ago
What's the importance of reproducible builds? Edit: found this nice overview by clicking a few links from the original post https://reproducible-builds.org/ https://reproducible-builds.org/
- johnmaguire2013 8y ago
  It allows you to verify that the source code was unaltered when the original build was produced. (By building it a second time with known good code.)
  - keanebean86 8y ago
    Does anyone diff public releases with builds done using a known good environment? It would be a lot of work but could expose malicious providers. Then again it might be hard to create an actual good environment considering all the software /firmware/hardware layers modern systems have.
    - orblivion 8y ago
      See here for an example: https://gitian.org/ https://gitian.org/. As I understand it's used to build a (the?) Bitcoin client.
    - kpcyrd 8y ago
      Yes, this is known as a rebuilder. There is still work to be done, but NYU was one of the organizations working on developing and running a rebuilder. The idea is that you pull buildinfo files from https://buildinfo.debian.net/ https://buildinfo.debian.net/, then try to verify them and if you got the same artifact you sign that you successfully verified this binary package. A user could then configure "I trust rebuilder X, Y, Z and I require that at least N have successfully verified the package" before installing it.
- bognition 8y ago
  Its extremely valuable for debugging. When you've got a bug that is in a specific build you can collect the set of dependencies that produced that build and BAM you've got an env that should replicate the bug.
iagooar 8y ago
For those of us unfamiliar with the term "reproducible", yet in the business of software - what does it mean and why is it desirable or needed?
- craftyguy 8y ago
  https://reproducible-builds.org/ https://reproducible-builds.org/
- rocqua 8y ago
  If you build the package correctly, you get the exact same binary. This allows users to verify that the binary packages you see were actually gotten by compiling the source code (presuming your compiler isn't compromised). This means that auditing the source-code is almost as good as auditing the package (the exception being compiler mistakes). Thus, we need less trust in the software packagers.
- scarejunba 8y ago
  Just wanted to add that besides the security gains there's a real performance gain. If you've got a slow-to-build codebase where you've made a small modification you can use a remote artifact cache so that all the parts that don't change just get downloaded from when someone built it. Then you have faster incremental compilation. You can do this with Bazel and Buck and with a few assumptions and some configuration even with Gradle.
  - chriswarbo 8y ago
    I think that's slightly different: if a build is reproducible, it means that the build products I get from my own machine are identical to those on a remote machine or cache (e.g. they have the same SHA hash). However, that doesn't speed anything up, since I don't know what the hash is unless I actually do the build or download the file (and then hash the result). If a cache used such hashes as IDs, I would only know which file to fetch once I've already got it! For such a cache to work there needs to be another mechanism for obtaining the IDs. As an example, Nix caches using the hash of build inputs (scripts, tarball URLs + hashes, etc.), not the build outputs. Since the build inputs are known before we do the build, we can combine these into a hash and query the cache. Since the whole point of using a cache is to avoid building things ourselves, we also need a separate mechanism to trust/verify what the cache has sent us (we could build it ourselves and compare, but then we might as well throw away the cache entirely!). Nix allows certain GPG keys to be trusted, and checks whether cached builds are signed. Since caching mechanisms don't make use of reproducibility, and verification mechanisms don't make use of reproducibility, such caches turn out not to require byte-for-byte reproducibility. All that's required is that plugging in the cached files gives a working result: in some cases that might be practically the same as byte-for-byte reproducibility (e.g. a C library compiled against certain ABIs at certain paths, etc.); others, like scripting languages, might work despite all sorts of shenanigans (e.g. a Python file might get converted into a different encoding; might get byte-compiled; might get optimised; might even get zipped!)
neatcoder 8y ago
What software currently has 100% reproducible builds?
- rbanffy 8y ago
  Any program with a build system designed in such way it doesn't introduce anything beyond the source code into the binary should be. If you build the same code on two different machines, using the same compiler, with the same options, then the generated binaries should be exactly the same.
  - biggerfisch 8y ago
    Until a build process starts naming things with timestamps, locales, etc. Just because the build is "source code only" doesn't mean it is deterministic.
    - rbanffy 8y ago
      That's why I wrote "it doesn't introduce anything beyond the source code into the binary". Unfortunately, I forgot to emphasize the anything. A build process that names things with timestamps or leaks your locale into the build configuration (or doesn't pin build-time dependency versions) will make the build depend on things other than the source code (both program and build settings) you made available. It may even be desirable for it to be non-reproductible - if, for instance, you want to use optimizations targeted to your specific system, then your build system will have to introduce the architecture information into the build process and your build will result in a unique binary that targets your own machine.
      - vishvananda 8y ago
        This is actually an annoying challenge of reproducible builds. In many cases it is actually useful to have a build timestamp, git sha, or build number available for debug output from the program. I've often gone as far as embedding a sha and/or timestamp into a file on export into a tgz which allows it to be reproducible from the tarfile, although builds directly out of source control would not be.
abdullahkhalids 8y ago
We need a body/service that verifies that the code-binary combo being distributed by a developer is actually reproducible. Because, most users don't have the time or resources to verify each package they use. But its useful to society if distributed packages are reproducible. So a trusted third party should provide this service. I would envisage it working in the following way. The developer submits links to their code and links to their package. The service builds from code and checks if the final package matches what is being distributed by the developer. If yes, it then publishes the hashes of the code and package, so users can quickly check that they are using a reproducibly built package from the correct source.
- Foxboron 8y ago
  This is being actively worked on and already has had quite a bit of work. This project is also submitted to Google summer of code under the Debian project. https://ssl.engineering.nyu.edu/blog/2019-01-18-in-toto-paris https://ssl.engineering.nyu.edu/blog/2019-01-18-in-toto-pari... https://salsa.debian.org/reproducible-builds/debian-rebuilder-setup https://salsa.debian.org/reproducible-builds/debian-rebuilde... https://github.com/in-toto/apt-transport-in-toto https://github.com/in-toto/apt-transport-in-toto https://reproducible-builds.org/docs/sharing-certifications/ https://reproducible-builds.org/docs/sharing-certifications/
- sigmar 8y ago
  The way F-Droid (android app distribution service) does this is all builds are done server-side (devs only update the publicly available source) and then anyone can quickly setup a verification server that builds+diffs the packages https://verification.f-droid.org https://verification.f-droid.org
jopsen 8y ago
Wow, I'm impressed by the consistent progress. There is so many things based on Debian and adding reproducibility in Debian is a massive security improvement. I have a lot of respect for people working on this, it must be hard to respect release freezes when you've been working on this for years.
fizixer 8y ago
Can anyone recommend a good way to install/maintain/remove new software on distros like Debian or Ubuntu LTS? (By 'new' I mean versions newer than the ones available through apt). I never figured out a maintainable scheme. Some of my packages are in /opt, some are in ~, some in ~/local, and some under GNU stow. And other than GNU stow's half-assed uninstall method, it's always a pain in the neck when I have to remove or upgrade any of these packages.
- andrenth 8y ago
  One thing that works many times is to download the deb source package from a newer distro and rebuild it in the desired system. If there’s no newer package available you can try to reuse the debian directory from the older source package to build from the newer source. Sometimes this works in the first attempt, but if you need this for a lot of packages, it’ll be a lot of work.
- JoshuaRLi 8y ago
  Why do you find GNU stow's uninstall method to be half-assed? Just curious.
trumped 8y ago
Might be good progress but it still sounds very low to me as I didn't know anything below 100% was possible... it sounds crazy to me (almost like something that was introduced to be able to inject backdoors undetected).
- TaylorAlexander 8y ago
  My guess is that making code reproducible involves some kind of change that hasn’t been applied or all of the code or build files.
- chriswarbo 8y ago
  Lots of problems come from things like timestamps, or race conditions in concurrent build systems giving slightly different bytes on disk. These generally aren't "trusting trust" level problems, since they do not and cannot affect program behaviour; but they do screw up things like digital signing, cryptographic hashes, etc. which are useful for automatically verifying that self-built artefacts are the same as distro-provided ones. These problems can also cascade, if component A embeds the hash of another component B, e.g. to verify that it's been given a correct version. If that hash comes from an unreproducible upstream, and building it ourselves gives a different hash, then we'll need to alter component A to use that new hash. That, in turn, changes the hash of component A, which might be referenced in some other component C, and so on.
- anth_anm 8y ago
  Nope, loads of build tools were never built with reproducibility in mind. Look at windows. Even if you fix the compiler and linker, you still non-reproducibility by design, the PE header contains a timestamp. People also like to stick non-reproducible stuff into builds directly, like timestamps. Compilers don't have any reason to lay down data in a specific order, so if they are threaded in the backend they just don't. IDL tools might stick in the timestamp of when a file was generated, for convenience. and on and on and on.
- jdblair 8y ago
  Every significant project I've worked on embedded the build host and build time in the resulting executable or firmware image. This was along with other static build information, like version number, compiler version and build flags. Once you make the sensible choice to include build time in the result you've broken reproducibility. Fixing this means tracking down every package that does this and removing the timestamp.
tofflos 8y ago
This seems to be the issue https://issues.apache.org/jira/browse/MNG-6276 https://issues.apache.org/jira/browse/MNG-6276 for reproducible builds with Maven. I'm not sure what the current status is given that all the issue links are closed. Go in and vote for it anyway. I'm sure some of the Debian packages are written in Java. ;-)
crb002 8y ago
Helps having Google as a client wanting reproducible builds.
cmurf 8y ago
I definitely understand the merit of making it super easy to verify, by simply hashing the build ISO. But that also shifts the burden of forcing deterministic results to the build process. I wonder if for a little more complexity in verification, if a lot less complexity would be needed on the build side? For example the ISO file system, typically ISO 9660 or UDF, will have a volume UUID from a random number. Sure you can code a flag for mkfs to specify a fixed number, that's easy. But then next that ISO typically contains a payload in the form of a squashfs file. And quite a bit of work on squashfs has happened to make sure file timestamps can be set to a known value. However, if one build process uses xz level 3 compression, and another build process uses xz level 7 compression - hashing will of course fail. The point is, be it inode UUIDs and timestamps and compression levels, there's a lot being measured that we don't really care about, just to have a simple verification method at the back end.
hathawsh 8y ago
I find it fascinating that, from a security standpoint, it's best for the static representation of a program (the executable and the package) to be perfectly predictable, while the dynamic representation of a program (once it's loaded into memory and running) should be as unpredictable as possible while maintaining correctness (using ASLR and other methods to slow down attacks on vulnerable code). I guess running a program is like setting off an explosive: you need to know exactly what you're handling before you release its power. Anyway, many kudos for this work!
- zachrose 8y ago
  What do you mean by "as unpredictable as possible"? From the perspective of an application developer I want that dynamic representation to also be predictable.
  - d33 8y ago
    https://en.wikipedia.org/wiki/Address_space_layout_randomization https://en.wikipedia.org/wiki/Address_space_layout_randomiza...
  - lallysingh 8y ago
    For someone who's found a way to inject data into your app, you don't want them to know the address of your symbols and data structures.
    - tedunangst 8y ago
      In other words, computer exploitation is basically hostile debugging.
      - yarrel 8y ago
        Enbugging. ;-)
      - lallysingh 8y ago
        Well, bugfinding :-)
jwiley 8y ago
My experience at Dave & Buster's is around 80% reproducible, so I think progress is possible.
- nodesocket 8y ago
  FYI, Debian codenames are all based off Toy Story characters. See https://unix.stackexchange.com/questions/222394/linux-debian-codenames https://unix.stackexchange.com/questions/222394/linux-debian...
getcrunk 8y ago
Eli5?

280 points by JNRowe 8y ago

JNRowe 8y ago

Slightly fluid numbers aside, half the packages for Buster still sounds like good progress to me.

e1ven 8y ago
It's certainly the right direction! I applaud them for pushing this. From the article, it looks like many of the others are reproducible on a code level, but the release system is using older binaries, which haven't been rebuilt yet. It's too bad it won't be fixed in time for this release, but bodes very well for future releases.

aruggirello 8y ago

BTW I was just looking at diffoscope - developed as part of the "reproducible builds" Debian project. https://try.diffoscope.org/ https://try.diffoscope.org/

peterkelly 8y ago

One has to step back and ask: "What made you decide to introduce non-determinism into your compilation process in the first place?"

dzdt 8y ago
If you aren't thinking about reproducibility things like putting timestamps in the binary seem reasonable. Which is what lots of tools did. Hence the problem.
jamieson-becker 8y ago
You don't really 'introduce non-determinism'. That's like saying, "why did you introduce entropy?" It's there. It happens, and it's difficult to get rid of in any complex system. https://en.wikipedia.org/wiki/Reproducible_builds https://en.wikipedia.org/wiki/Reproducible_builds https://wiki.debian.org/ReproducibleBuilds https://wiki.debian.org/ReproducibleBuilds https://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html https://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html Debian was one of the first large distributions to use a build farm for many different architectures. Maintaining deterministic builds (even as far as just file timestamps) in a distributed environment is very challenging.
pm215 8y ago
Nobody sat down and said "yes, I'll make this nondeterministic". If you look at the wiki page's sampling of different issues -- https://wiki.debian.org/ReproducibleBuilds/Howto#Identified_problems.2C_and_possible_solutions https://wiki.debian.org/ReproducibleBuilds/Howto#Identified_... -- you'll see it's a mix of various things: * output is deterministic but dependent on some aspect of the build environment (locale, hostname, etc) * output is accidentally non-deterministic (eg "we put all the .html files into a tarball with a shell glob pattern, which gets you an order dependent on your filesystem implementation and the phase of the moon") * a wide array of "output contains a timestamp" issues "Let's put the timestamp into our version string/a generated file so the user knows when it was built" is a really common thing, and it seemed like purely a nice convenience feature until the concept of 100% binary-reproducible builds became a current concern.
- jlarocco 8y ago
  Except for the first one, the last three are great examples of why I don't care about 100% binary reproducible builds as an end user. Knowing when something was built is more valuable to me than a reproducible build. Seems like a fad triumphing over common sense.
  - peterwwillis 8y ago
    Users often don't care about things until they become a problem. For example, most users don't care how their password is stored on a server... until it gets compromised. Also, timestamps in output usually indicate source code age, not build date. I have a package with '2015' in the version but it was built in 2018.
    - jlarocco 8y ago
      That's a condescending non-answer. We're not talking about passwords in a database, we're talking about reproducible builds. I already trust Debian, so it's irrelevant to me, as a user, if their builds are 100% reproducible. Can you explain why I should feel differently? Although I don't think it's a valuable thing to spend time on, it's not my time, so I honestly don't care too much.
scarejunba 8y ago
This is true and it's also the model I use to avoid bugs. Any time I'd add a bug to the codebase, I just look at it and decide not to write the bug. In this way, no bugs are ever introduced.

badfrog 8y ago

What's the importance of reproducible builds? Edit: found this nice overview by clicking a few links from the original post https://reproducible-builds.org/ https://reproducible-builds.org/

johnmaguire2013 8y ago
It allows you to verify that the source code was unaltered when the original build was produced. (By building it a second time with known good code.)
- keanebean86 8y ago
  Does anyone diff public releases with builds done using a known good environment? It would be a lot of work but could expose malicious providers. Then again it might be hard to create an actual good environment considering all the software /firmware/hardware layers modern systems have.
  - orblivion 8y ago
    See here for an example: https://gitian.org/ https://gitian.org/. As I understand it's used to build a (the?) Bitcoin client.
  - kpcyrd 8y ago
    Yes, this is known as a rebuilder. There is still work to be done, but NYU was one of the organizations working on developing and running a rebuilder. The idea is that you pull buildinfo files from https://buildinfo.debian.net/ https://buildinfo.debian.net/, then try to verify them and if you got the same artifact you sign that you successfully verified this binary package. A user could then configure "I trust rebuilder X, Y, Z and I require that at least N have successfully verified the package" before installing it.
bognition 8y ago
Its extremely valuable for debugging. When you've got a bug that is in a specific build you can collect the set of dependencies that produced that build and BAM you've got an env that should replicate the bug.

iagooar 8y ago

For those of us unfamiliar with the term "reproducible", yet in the business of software - what does it mean and why is it desirable or needed?

craftyguy 8y ago
https://reproducible-builds.org/ https://reproducible-builds.org/
rocqua 8y ago
If you build the package correctly, you get the exact same binary. This allows users to verify that the binary packages you see were actually gotten by compiling the source code (presuming your compiler isn't compromised). This means that auditing the source-code is almost as good as auditing the package (the exception being compiler mistakes). Thus, we need less trust in the software packagers.
scarejunba 8y ago
Just wanted to add that besides the security gains there's a real performance gain. If you've got a slow-to-build codebase where you've made a small modification you can use a remote artifact cache so that all the parts that don't change just get downloaded from when someone built it. Then you have faster incremental compilation. You can do this with Bazel and Buck and with a few assumptions and some configuration even with Gradle.
- chriswarbo 8y ago
  I think that's slightly different: if a build is reproducible, it means that the build products I get from my own machine are identical to those on a remote machine or cache (e.g. they have the same SHA hash). However, that doesn't speed anything up, since I don't know what the hash is unless I actually do the build or download the file (and then hash the result). If a cache used such hashes as IDs, I would only know which file to fetch once I've already got it! For such a cache to work there needs to be another mechanism for obtaining the IDs. As an example, Nix caches using the hash of build inputs (scripts, tarball URLs + hashes, etc.), not the build outputs. Since the build inputs are known before we do the build, we can combine these into a hash and query the cache. Since the whole point of using a cache is to avoid building things ourselves, we also need a separate mechanism to trust/verify what the cache has sent us (we could build it ourselves and compare, but then we might as well throw away the cache entirely!). Nix allows certain GPG keys to be trusted, and checks whether cached builds are signed. Since caching mechanisms don't make use of reproducibility, and verification mechanisms don't make use of reproducibility, such caches turn out not to require byte-for-byte reproducibility. All that's required is that plugging in the cached files gives a working result: in some cases that might be practically the same as byte-for-byte reproducibility (e.g. a C library compiled against certain ABIs at certain paths, etc.); others, like scripting languages, might work despite all sorts of shenanigans (e.g. a Python file might get converted into a different encoding; might get byte-compiled; might get optimised; might even get zipped!)

neatcoder 8y ago

What software currently has 100% reproducible builds?

rbanffy 8y ago
Any program with a build system designed in such way it doesn't introduce anything beyond the source code into the binary should be. If you build the same code on two different machines, using the same compiler, with the same options, then the generated binaries should be exactly the same.
- biggerfisch 8y ago
  Until a build process starts naming things with timestamps, locales, etc. Just because the build is "source code only" doesn't mean it is deterministic.
  - rbanffy 8y ago
    That's why I wrote "it doesn't introduce anything beyond the source code into the binary". Unfortunately, I forgot to emphasize the anything. A build process that names things with timestamps or leaks your locale into the build configuration (or doesn't pin build-time dependency versions) will make the build depend on things other than the source code (both program and build settings) you made available. It may even be desirable for it to be non-reproductible - if, for instance, you want to use optimizations targeted to your specific system, then your build system will have to introduce the architecture information into the build process and your build will result in a unique binary that targets your own machine.
    - vishvananda 8y ago
      This is actually an annoying challenge of reproducible builds. In many cases it is actually useful to have a build timestamp, git sha, or build number available for debug output from the program. I've often gone as far as embedding a sha and/or timestamp into a file on export into a tgz which allows it to be reproducible from the tarfile, although builds directly out of source control would not be.

abdullahkhalids 8y ago

We need a body/service that verifies that the code-binary combo being distributed by a developer is actually reproducible. Because, most users don't have the time or resources to verify each package they use. But its useful to society if distributed packages are reproducible. So a trusted third party should provide this service. I would envisage it working in the following way. The developer submits links to their code and links to their package. The service builds from code and checks if the final package matches what is being distributed by the developer. If yes, it then publishes the hashes of the code and package, so users can quickly check that they are using a reproducibly built package from the correct source.

Foxboron 8y ago
This is being actively worked on and already has had quite a bit of work. This project is also submitted to Google summer of code under the Debian project. https://ssl.engineering.nyu.edu/blog/2019-01-18-in-toto-paris https://ssl.engineering.nyu.edu/blog/2019-01-18-in-toto-pari... https://salsa.debian.org/reproducible-builds/debian-rebuilder-setup https://salsa.debian.org/reproducible-builds/debian-rebuilde... https://github.com/in-toto/apt-transport-in-toto https://github.com/in-toto/apt-transport-in-toto https://reproducible-builds.org/docs/sharing-certifications/ https://reproducible-builds.org/docs/sharing-certifications/
sigmar 8y ago
The way F-Droid (android app distribution service) does this is all builds are done server-side (devs only update the publicly available source) and then anyone can quickly setup a verification server that builds+diffs the packages https://verification.f-droid.org https://verification.f-droid.org

jopsen 8y ago

Wow, I'm impressed by the consistent progress. There is so many things based on Debian and adding reproducibility in Debian is a massive security improvement. I have a lot of respect for people working on this, it must be hard to respect release freezes when you've been working on this for years.

fizixer 8y ago

Can anyone recommend a good way to install/maintain/remove new software on distros like Debian or Ubuntu LTS? (By 'new' I mean versions newer than the ones available through apt). I never figured out a maintainable scheme. Some of my packages are in /opt, some are in ~, some in ~/local, and some under GNU stow. And other than GNU stow's half-assed uninstall method, it's always a pain in the neck when I have to remove or upgrade any of these packages.

andrenth 8y ago
One thing that works many times is to download the deb source package from a newer distro and rebuild it in the desired system. If there’s no newer package available you can try to reuse the debian directory from the older source package to build from the newer source. Sometimes this works in the first attempt, but if you need this for a lot of packages, it’ll be a lot of work.
JoshuaRLi 8y ago
Why do you find GNU stow's uninstall method to be half-assed? Just curious.

trumped 8y ago

Might be good progress but it still sounds very low to me as I didn't know anything below 100% was possible... it sounds crazy to me (almost like something that was introduced to be able to inject backdoors undetected).

TaylorAlexander 8y ago
My guess is that making code reproducible involves some kind of change that hasn’t been applied or all of the code or build files.
chriswarbo 8y ago
Lots of problems come from things like timestamps, or race conditions in concurrent build systems giving slightly different bytes on disk. These generally aren't "trusting trust" level problems, since they do not and cannot affect program behaviour; but they do screw up things like digital signing, cryptographic hashes, etc. which are useful for automatically verifying that self-built artefacts are the same as distro-provided ones. These problems can also cascade, if component A embeds the hash of another component B, e.g. to verify that it's been given a correct version. If that hash comes from an unreproducible upstream, and building it ourselves gives a different hash, then we'll need to alter component A to use that new hash. That, in turn, changes the hash of component A, which might be referenced in some other component C, and so on.
anth_anm 8y ago
Nope, loads of build tools were never built with reproducibility in mind. Look at windows. Even if you fix the compiler and linker, you still non-reproducibility by design, the PE header contains a timestamp. People also like to stick non-reproducible stuff into builds directly, like timestamps. Compilers don't have any reason to lay down data in a specific order, so if they are threaded in the backend they just don't. IDL tools might stick in the timestamp of when a file was generated, for convenience. and on and on and on.
jdblair 8y ago
Every significant project I've worked on embedded the build host and build time in the resulting executable or firmware image. This was along with other static build information, like version number, compiler version and build flags. Once you make the sensible choice to include build time in the result you've broken reproducibility. Fixing this means tracking down every package that does this and removing the timestamp.

tofflos 8y ago

This seems to be the issue https://issues.apache.org/jira/browse/MNG-6276 https://issues.apache.org/jira/browse/MNG-6276 for reproducible builds with Maven. I'm not sure what the current status is given that all the issue links are closed. Go in and vote for it anyway. I'm sure some of the Debian packages are written in Java. ;-)

crb002 8y ago

Helps having Google as a client wanting reproducible builds.

cmurf 8y ago

I definitely understand the merit of making it super easy to verify, by simply hashing the build ISO. But that also shifts the burden of forcing deterministic results to the build process. I wonder if for a little more complexity in verification, if a lot less complexity would be needed on the build side? For example the ISO file system, typically ISO 9660 or UDF, will have a volume UUID from a random number. Sure you can code a flag for mkfs to specify a fixed number, that's easy. But then next that ISO typically contains a payload in the form of a squashfs file. And quite a bit of work on squashfs has happened to make sure file timestamps can be set to a known value. However, if one build process uses xz level 3 compression, and another build process uses xz level 7 compression - hashing will of course fail. The point is, be it inode UUIDs and timestamps and compression levels, there's a lot being measured that we don't really care about, just to have a simple verification method at the back end.

hathawsh 8y ago

I find it fascinating that, from a security standpoint, it's best for the static representation of a program (the executable and the package) to be perfectly predictable, while the dynamic representation of a program (once it's loaded into memory and running) should be as unpredictable as possible while maintaining correctness (using ASLR and other methods to slow down attacks on vulnerable code). I guess running a program is like setting off an explosive: you need to know exactly what you're handling before you release its power. Anyway, many kudos for this work!

zachrose 8y ago
What do you mean by "as unpredictable as possible"? From the perspective of an application developer I want that dynamic representation to also be predictable.
- d33 8y ago
  https://en.wikipedia.org/wiki/Address_space_layout_randomization https://en.wikipedia.org/wiki/Address_space_layout_randomiza...
- lallysingh 8y ago
  For someone who's found a way to inject data into your app, you don't want them to know the address of your symbols and data structures.
  - tedunangst 8y ago
    In other words, computer exploitation is basically hostile debugging.
    - yarrel 8y ago
      Enbugging. ;-)
    - lallysingh 8y ago
      Well, bugfinding :-)

jwiley 8y ago

My experience at Dave & Buster's is around 80% reproducible, so I think progress is possible.

nodesocket 8y ago
FYI, Debian codenames are all based off Toy Story characters. See https://unix.stackexchange.com/questions/222394/linux-debian-codenames https://unix.stackexchange.com/questions/222394/linux-debian...

getcrunk 8y ago

Eli5?