5 ms·

Reproducibility changes everything, because nothing changes. We go from shamans chanting incantations over a blessed code base to a mathematical function with a

Reproducibility changes everything, because nothing changes. We go from shamans chanting incantations over a blessed code base to a mathematical function with an algebra of system composition. Let me give you the simplest example, when builds are reproducible you don't need package repositories, you need build caches. All the problems with maintaining a repository (save bandwidth) evaporate.

okanat 2y ago
> Let me give you the simplest example, when builds are reproducible you don't need package repositories, you need build caches. The type of reproducibility is different here. What you mention is possible via a stable compiler ABI already. However one needs to keep the source code the same. Without a stable compiler ABI, you may or may not fix it depending on what the compiler does. The goal of reproducible builds is removing sources of environment-dependent behavior at the build level instead of the compiler level. So given all the same dependencies and same build commands your binaries should match wherever and whenever you compile them. The distros and software developers also made a huge effort to remove any kind of environment-dependent commands. Different distros still have differences in the build commands they issue and the set of dependencies they enable. The space to cache each individual possible output would be enormous and impractical. So you would still need repositories.
- anotherhue 2y ago
  All true, but I can't not take this opportunity to shill NixOS which has meaningfully addressed many of these issues, and is indeed spending impractical amounts of money storing build outputs ($10k/m). https://discourse.nixos.org/t/the-nixos-foundations-call-to-action-s3-costs-require-community-support/28672 https://discourse.nixos.org/t/the-nixos-foundations-call-to-... It is absolutely better to remove build entropy at the source code stage, but until all software is written that way there are a few build-environment tricks we can use along the way.
  - zelphirkalt 2y ago
    I highly doubt, that we will get anywhere close to developers understanding and valuing reproducibility any time soon. Maybe in 20y or so. Basically outside of the corners like Nix, Guix and maybe a few random people in discussions about issues of package managers, I have not met anyone knowing how to and caring about reproducibility. Meanwhile I enjoy setting up my own GNU Guile projects with a Makefile that sets up a reproducible Guix shell in which my project is run, so that I get the same result on various devices, only needing to issue a single easily memorized or discoverable Makefile target call. Most developers I met don't know how to set something like this up. Provided no one messes with guix package manager commits and guix infrastructure still exists, my projects will run in 10y just like they do today, with reproducible result. Neat. Recently I have taken some time to have this for Ocaml as well. Took some asking on the mailing list, but now works. No need to have anything installed prior to running the Makefile target other than Make and Guix package manager.
    - anotherhue 2y ago
      I assume you tried Flakes also? Does Guile have a better approach to this problem (other than the general language difference (everyone hate nix)). Edit: Worth looking at https://github.com/numtide/devshell https://github.com/numtide/devshell
      - zelphirkalt 2y ago
        I have not tried Flakes. Guile's main way of installing packages, I believe, is using Guix package manager. And Guix is written using Guile.
    - patrakov 2y ago
      > Basically outside of the corners like Nix, Guix and maybe a few random people in discussions about issues of package managers, I have not met anyone knowing how to and caring about reproducibility. And I did. He was a developer of a purported secure messaging app. The question that changed his mind was: "How are your users, who surely audited your sources and found no trojans hiding there, going to know that you are not distributing trojaned official binary builds?"
- jimmaswell 2y ago
  > The space to cache each individual possible output would be enormous and impractical. So you would still need repositories. I've been using Gentoo for the past few weeks and this was my thought. There are so many ways to compile packages depending on your needs. It is practical to make binary sets for a few common configurations that are good enough for most people for each distro, though. (Installing and using Gentoo has been an incredible learning experience. You have to go into it with the desire to take long tangents filling the gaps in your knowledge re: shared libraries, kernel modules, your init system of choice, gcc, your windowing system of choice, bootloaders, bash, etc. I feel like I've made more of a quantum leap in my Linux skills the past month than I have in years, and it's been quite fun and rewarding.) > Different distros still have differences in the build commands they issue and the set of dependencies they enable. Would it even make sense in theory for this not to be the case? What is a Linux distro but a set of programs, libraries, and environment choices chosen to be run on the Linux kernel?
- kbaker 2y ago
  The Yocto Project (from the embedded space) has reproducibility as one of its goals. Since everything is built from source and from scratch, even the build toolchains, it is not too big of a step. Seems like Yocto would make the base for a good general purpose desktop distro (if there is not one out there already.)
  - transpute 2y ago
    Yocto still has dependencies on a bootstrap build distro. Stagex + Yocto would be fully reproducible from a small seed. > Yocto would make the base for a good general purpose desktop distro YP has been working on a public binary reference cache/distro.
  - rcxdude 2y ago
    Yocto was kinda doing the nix thing for a while before nix existed, but basically by slowly growing the capabilities in an ad-hoc fashion instead of working on it from first principles. It's resulted in a bit of a mess (it's an unholy mix of a custom functional-ish programming language grown out of a config file format, python, and bash, with a ludicrous capacity for action-at-distance) but there's still not really anything else like it.
    - transpute 2y ago
      > unholy mix ... ludicrous capacity for action-at-distance OE recipe-grade concision! This summary belongs in YP docs, wikipedia and LLM IDE warnings.
BiteCode_dev 2y ago
You mean you don't need dep resolution, signatures and moderation? How so?
- anotherhue 2y ago
  Good questions: 1. If the deps are also themselves reproducible then you refer to a fixed point version of them and (at least for nix) the package manager works out the rest. 2. Signatures are a trust mechanism, if a cache feeds you bad data in response to a query then that's absolutely an issue, but since there can be multiple caches (or your own local spot-checks) it becomes easier to detect if a cache is returning bad data. A hyper-targetted attack would still get you unless you decide to manually build certain packages, but that's no different than existing repos. Manually building might sound impractical but it doesn't actually take that long, probably less than a day for a desktop environment, which might acceptable in a high-trust environment if amortized. I should add that the process is fully automatic, it just takes longer than using a cache. 3. Moderation I don't have a good answer to, anyone can run an apt server, or publish a flake.nix file to a repo. Some would say it is censorship resistant.
- LtWorf 2y ago
  He hasn't understood what reproducible builds are.
XorNot 2y ago
This doesn't work at all: a build cache is meaningless in this context because the only thing you can do is rebuild the code to verify that what is in the cache is what comes from the code you expect...or to bootstrap the whole compiler chain, then use cryptographic signatures to chain trust from a matching compiler hash up to all the dependent outputs. It certainly doesn't change the nature of packaging in any real way in that case.
- anotherhue 2y ago
  Others can rebuild the chain and the results can be compared. Without reproducibility there is no concept of comparison.
  - XorNot 2y ago
    Which is still a cryptographic trust system - i.e. who are the others who built it and are they suitably independent or working from similar sources?
    - vlovich123 2y ago
      It turns out that conspiracies are really hard to maintain the larger the network gets due to network effects. All it takes is one guy to run it, get different results and go “guys I’m noticing something weird” to start the ball rolling. As a proof point, look at what happened to the attack on ssh by way of a xz utils back door.
    - gosub100 2y ago
      It's not just detecting malicious actions, binary reproducability helps solve "works on my machine" bugs because you'll get a different hash if you, say, link against a library you didn't think was being selected.
blueflow 2y ago
Do they evaporate? My impression was that they got shifted to the package recipes. Like Nix having to take the same considerations with channels.
kazinator 2y ago
But, same thing when you never reproduce builds (just build once and treat as gold master, same thing) So that's a minor value. Build reproducibility has other values, like the virtue of basic determinism. That answers the question of why would we want to reproduce a build? If you /can't/ build the exact same thing twice, down to the bit, how do you know that, say, a compiler isn't doing something funny based on an uninitialized variable? To check reproducible builds against regression, we can't always be caching. Someone has to actually build everything at least twice (ideally more times) and verify that they match. And this has to be done forever; if you don't build multiple times and check, you will not catch a regression in reproducibility.

okanat 2y ago

> Let me give you the simplest example, when builds are reproducible you don't need package repositories, you need build caches. The type of reproducibility is different here. What you mention is possible via a stable compiler ABI already. However one needs to keep the source code the same. Without a stable compiler ABI, you may or may not fix it depending on what the compiler does. The goal of reproducible builds is removing sources of environment-dependent behavior at the build level instead of the compiler level. So given all the same dependencies and same build commands your binaries should match wherever and whenever you compile them. The distros and software developers also made a huge effort to remove any kind of environment-dependent commands. Different distros still have differences in the build commands they issue and the set of dependencies they enable. The space to cache each individual possible output would be enormous and impractical. So you would still need repositories.

anotherhue 2y ago
All true, but I can't not take this opportunity to shill NixOS which has meaningfully addressed many of these issues, and is indeed spending impractical amounts of money storing build outputs ($10k/m). https://discourse.nixos.org/t/the-nixos-foundations-call-to-action-s3-costs-require-community-support/28672 https://discourse.nixos.org/t/the-nixos-foundations-call-to-... It is absolutely better to remove build entropy at the source code stage, but until all software is written that way there are a few build-environment tricks we can use along the way.
- zelphirkalt 2y ago
  I highly doubt, that we will get anywhere close to developers understanding and valuing reproducibility any time soon. Maybe in 20y or so. Basically outside of the corners like Nix, Guix and maybe a few random people in discussions about issues of package managers, I have not met anyone knowing how to and caring about reproducibility. Meanwhile I enjoy setting up my own GNU Guile projects with a Makefile that sets up a reproducible Guix shell in which my project is run, so that I get the same result on various devices, only needing to issue a single easily memorized or discoverable Makefile target call. Most developers I met don't know how to set something like this up. Provided no one messes with guix package manager commits and guix infrastructure still exists, my projects will run in 10y just like they do today, with reproducible result. Neat. Recently I have taken some time to have this for Ocaml as well. Took some asking on the mailing list, but now works. No need to have anything installed prior to running the Makefile target other than Make and Guix package manager.
  - anotherhue 2y ago
    I assume you tried Flakes also? Does Guile have a better approach to this problem (other than the general language difference (everyone hate nix)). Edit: Worth looking at https://github.com/numtide/devshell https://github.com/numtide/devshell
    - zelphirkalt 2y ago
      I have not tried Flakes. Guile's main way of installing packages, I believe, is using Guix package manager. And Guix is written using Guile.
  - patrakov 2y ago
    > Basically outside of the corners like Nix, Guix and maybe a few random people in discussions about issues of package managers, I have not met anyone knowing how to and caring about reproducibility. And I did. He was a developer of a purported secure messaging app. The question that changed his mind was: "How are your users, who surely audited your sources and found no trojans hiding there, going to know that you are not distributing trojaned official binary builds?"
jimmaswell 2y ago
> The space to cache each individual possible output would be enormous and impractical. So you would still need repositories. I've been using Gentoo for the past few weeks and this was my thought. There are so many ways to compile packages depending on your needs. It is practical to make binary sets for a few common configurations that are good enough for most people for each distro, though. (Installing and using Gentoo has been an incredible learning experience. You have to go into it with the desire to take long tangents filling the gaps in your knowledge re: shared libraries, kernel modules, your init system of choice, gcc, your windowing system of choice, bootloaders, bash, etc. I feel like I've made more of a quantum leap in my Linux skills the past month than I have in years, and it's been quite fun and rewarding.) > Different distros still have differences in the build commands they issue and the set of dependencies they enable. Would it even make sense in theory for this not to be the case? What is a Linux distro but a set of programs, libraries, and environment choices chosen to be run on the Linux kernel?
kbaker 2y ago
The Yocto Project (from the embedded space) has reproducibility as one of its goals. Since everything is built from source and from scratch, even the build toolchains, it is not too big of a step. Seems like Yocto would make the base for a good general purpose desktop distro (if there is not one out there already.)
- transpute 2y ago
  Yocto still has dependencies on a bootstrap build distro. Stagex + Yocto would be fully reproducible from a small seed. > Yocto would make the base for a good general purpose desktop distro YP has been working on a public binary reference cache/distro.
- rcxdude 2y ago
  Yocto was kinda doing the nix thing for a while before nix existed, but basically by slowly growing the capabilities in an ad-hoc fashion instead of working on it from first principles. It's resulted in a bit of a mess (it's an unholy mix of a custom functional-ish programming language grown out of a config file format, python, and bash, with a ludicrous capacity for action-at-distance) but there's still not really anything else like it.
  - transpute 2y ago
    > unholy mix ... ludicrous capacity for action-at-distance OE recipe-grade concision! This summary belongs in YP docs, wikipedia and LLM IDE warnings.

BiteCode_dev 2y ago

You mean you don't need dep resolution, signatures and moderation? How so?

anotherhue 2y ago
Good questions: 1. If the deps are also themselves reproducible then you refer to a fixed point version of them and (at least for nix) the package manager works out the rest. 2. Signatures are a trust mechanism, if a cache feeds you bad data in response to a query then that's absolutely an issue, but since there can be multiple caches (or your own local spot-checks) it becomes easier to detect if a cache is returning bad data. A hyper-targetted attack would still get you unless you decide to manually build certain packages, but that's no different than existing repos. Manually building might sound impractical but it doesn't actually take that long, probably less than a day for a desktop environment, which might acceptable in a high-trust environment if amortized. I should add that the process is fully automatic, it just takes longer than using a cache. 3. Moderation I don't have a good answer to, anyone can run an apt server, or publish a flake.nix file to a repo. Some would say it is censorship resistant.
LtWorf 2y ago
He hasn't understood what reproducible builds are.

XorNot 2y ago

This doesn't work at all: a build cache is meaningless in this context because the only thing you can do is rebuild the code to verify that what is in the cache is what comes from the code you expect...or to bootstrap the whole compiler chain, then use cryptographic signatures to chain trust from a matching compiler hash up to all the dependent outputs. It certainly doesn't change the nature of packaging in any real way in that case.

anotherhue 2y ago
Others can rebuild the chain and the results can be compared. Without reproducibility there is no concept of comparison.
- XorNot 2y ago
  Which is still a cryptographic trust system - i.e. who are the others who built it and are they suitably independent or working from similar sources?
  - vlovich123 2y ago
    It turns out that conspiracies are really hard to maintain the larger the network gets due to network effects. All it takes is one guy to run it, get different results and go “guys I’m noticing something weird” to start the ball rolling. As a proof point, look at what happened to the attack on ssh by way of a xz utils back door.
  - gosub100 2y ago
    It's not just detecting malicious actions, binary reproducability helps solve "works on my machine" bugs because you'll get a different hash if you, say, link against a library you didn't think was being selected.

blueflow 2y ago

Do they evaporate? My impression was that they got shifted to the package recipes. Like Nix having to take the same considerations with channels.

kazinator 2y ago

But, same thing when you never reproduce builds (just build once and treat as gold master, same thing) So that's a minor value. Build reproducibility has other values, like the virtue of basic determinism. That answers the question of why would we want to reproduce a build? If you /can't/ build the exact same thing twice, down to the bit, how do you know that, say, a compiler isn't doing something funny based on an uninitialized variable? To check reproducible builds against regression, we can't always be caching. Someone has to actually build everything at least twice (ideally more times) and verify that they match. And this has to be done forever; if you don't build multiple times and check, you will not catch a regression in reproducibility.