5 ms·

Is my package reproducible yet?

80 points by pabs3 4y ago

stoplying1 4y ago
(doesn't list or use the package manager most likely to already be reproducibile)
- yjftsjthsd-h 4y ago
  ... Huh. Yeah, that's a really short list of distros, and omitting nix and guix seems like a weird thing in this space.
  - pabs3 4y ago
    Seems Nix/Guix are tracked outside of the Reproducible Builds infrastructure, but the site only lists those within that infra. https://reproducible-builds.org/citests/ https://reproducible-builds.org/citests/ https://r13y.com/ https://r13y.com/ https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/package-reproducibility https://data.guix.gnu.org/repository/1/branch/master/latest-...
- Foxboron 4y ago
  Which package manager would be more reproducible than pacman and Arch? We hover around 85% and 89% reproducible packages across all our packages and no distro, to my knowledge, is close to anything like this. https://reproducible.archlinux.org/ https://reproducible.archlinux.org/
  - kaba0 4y ago
    Nix?
    - Foxboron 4y ago
      It's not. It's marketing. The focus of Nix is reproducability in the same wain as docker: Same dependencies for the same behavior. They can't guarantee bit-for-bit reproducability any more then your average distribution. Arch Linux has a rebuilder setup and you can safely reproduce between 85% and 89% of the repository packages. Nix has never done such a thing, and would be far away from any comparable numbers.
      - kaba0 4y ago
        This is literally linked in a reply to you: https://r13y.com/ https://r13y.com/ Nix is 99.77% reproducible, so come on. It’s not marketing, nix just properly solves the problem for the first time ever. Guix is another example, it is more into “bootstrappability” though, so packages can be built from source all the way down to a few kilobyte C-subset compiler written in assembly. Very cool!
        Foxboron 4y ago
        >Nix is 99.77% reproducible, so come on. It’s not marketing, nix just properly solves the problem for the first time ever. You haven't read r13y properly. >1733 out of 1737 (99.77%) paths in the nixos.iso_minimal.x86_64-linux installation image are reproducible! 600 packages becomes 1737 paths which are the inputs used to generate the image. Several of the compared paths are just static files which doesn't change, so the "path to package" comparison doesn't work either. EDIT: An example set of stuff it attempts to compare can be found here: https://paste.xinu.at/TmVM0y5FBDWX/ https://paste.xinu.at/TmVM0y5FBDWX/ Taken from the build here: https://buildkite.com/grahamc/r13y-dot-com/builds/1149#01813950-9819-49f6-842f-b8e0aa325e1b https://buildkite.com/grahamc/r13y-dot-com/builds/1149#01813... They have not solved the problem.
        kaba0 4y ago
        It’s almost as if no matter how reproducible your dependencies are, if the given build process/tool wants to play dirty it will. I really don’t see how arch fairs any better here, care to elaborate? While it is a different problem, my litmus test for package managers is installing gnome a (the whole group) and then seeing whether I can transition to kde or vice versa without a bunch of dependencies lingering around forever. Pacman, apt, etc. all fail terribly here.
        Foxboron 4y ago
        > I really don’t see how arch fairs any better here, care to elaborate? Arch has spent a considerably amount of time getting to the figure it has and is in the same position as NixOS to maintain reproducible builds. The claim that NixOS solves this, or is uniquely dispositioned to solve this, is the false. That is what I'm effectively trying to point out here. >While it is a different problem, my litmus test for package managers is installing gnome a (the whole group) and then seeing whether I can transition to kde or vice versa without a bunch of dependencies lingering around forever. Pacman, apt, etc. all fail terribly here. That is not what Reproducible Builds is about.
pooper 4y ago
The "weather" icons sunny vs raining vs clock is a blast from the (Jenkins?) past :) For example: https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/ffmpeg.html https://tests.reproducible-builds.org/debian/rb-pkg/unstable...
- pabs3 4y ago
  The Reproducible Builds test infra indeed runs on Jenkins, specifically the jenkins.debian.net service.
stopdropnhotpot 4y ago
encryptluks2 4y ago
I'm using a hook for pacman on Arch to verify reproducible builds but one issue I've found is it slows down package management substantially. Has anyone else encountered this and is there a solution for verification without slowing down the package manager?
- aew4ytasghe5 4y ago
  What are you describing here, compile time for a local verification? If so, how would you imagine that could be sped up? - If the distro is working on reproducibility, they should have a dashboard somewhere with reproducibility status. For arch that would be https://reproducible.archlinux.org/ https://reproducible.archlinux.org/ It would be done in their CI and there should be no reason for you do also manually verify this, unless you are a very targeted individual and you think its reasonable that arch linux infrastructure is hacked to target you. Basically, either you trust your distro or you don't. If you are running local verifications of reproducibility, I am guessing you are not. So consider switching to a distro that you do trust.
  - encryptluks2 4y ago
    No, not compile time... just verifying binary transparency: https://github.com/kpcyrd/pacman-bintrans https://github.com/kpcyrd/pacman-bintrans
defrost 4y ago
I'm loving the related diffoscope utility (new to me) [1] https://diffoscope.org/ https://diffoscope.org/ It did a fantastic job of highlighting the few changes in a tree of XML "one liner" files (each file one ~ 25 Kb text line).
- pabs3 4y ago
  BTW, the latest version now checks if two XML files are identical except for ordering differences.
  - defrost 4y ago
    Yep - that was the killer feature that sold me! :-)
freeqaz 4y ago
One of my biggest pet peeves with NPM is that the packages aren't reproducible. Even if somebody links a package to their Git repo, who knows what commit correlates to a given release? Even if somebody pushes a "tag" for the release, there is still no guarantee that it's actually the same code. (especially if there is any type of build step.) I'm not 100% certain, but I believe this is an area where Golang is doing better (code references a Git repo directly). Are there any other languages that focus on "reproducible builds" for packages? (Ideally via Git) Edit: I just want to add that this is a major frustration for me with Java too. Everything on Maven barely has any metadata to go back and audit the source code!
- vbezhenar 4y ago
  What's wrong with maven? You have version in pom.xml, you can retrieve corresponding git tag and audit it.
  - chriswarbo 4y ago
    Version numbers are just a form of documentation, giving the developer's best guess of whether the API broke or not. In particular, Maven's use of groupId:artifactId:version is inadequate as an identifier, since many artifacts can have this same "identifier", e.g. - Making changes without updating the version number (bad practice, but easily done) - Building with different versions of dependencies. Maven allows version ranges, and resolves them based on the contents of third-party servers, so this can change over time even if we're building from the same git commit. - Building with different property settings, e.g. due to a different profile getting 'activated' due to some ambient aspect of the system (OS, presence/absence of some file path, env vars, etc.). - Building with different versions of Maven, Java, etc. - The contents of Maven's cache also affects builds. If a file path exists in the cache, Maven will use it regardless of its contents. Also, undeclared dependencies can sometimes be picked up from the cache, which would otherwise fail if they weren't in the cache. Maven also uses that scheme as a location for fetching artifacts, which is not only inadequate but also insecure (we can put arbitrary code at a certain path and Maven will happily fetch it). Maven does perform SHA512 verification, but those hashes are fetched from the same location as the artifacts, so subverting one allows subverting both. Also, in case you weren't aware, git tags are just a convenience. Commit IDs are SHA1 hashes, so they're unforgeable and tamper-proof; whereas git tags can be deleted and recreated to point at anything. FYI at my workplace we avoid some of these issues by: - Performing all builds using Nix, which uses a sandbox with versions of Maven, JVM, etc. specified with their SHA256 hashes. This avoids differences due to tooling, profiles, env vars, caches, etc. - All builds are performed without network access. This prevents Maven resolving version numbers differently. - The SHA256 of the build environment (including all tooling, env vars, properties, source code, etc.) gets appended to the version number, which distinguishes these different artifacts from each other. - Our projects specify the expected hash of their Maven dependencies (essentially like a lock file), which avoids having to trust third-parties for the hashes (a trust-on-first-use policy).
zuzuleinen 4y ago
Using Nix[0] should solve the reproducibility of a package. We use its package manager for bob[1] to achieve reproducible builds for projects. [0] https://nixos.org/guides/how-nix-works.html https://nixos.org/guides/how-nix-works.html [1] https://bob.build/ https://bob.build/
- pabs3 4y ago
  Unfortunately, not 100% of Nix packages are reproducible, so using it doesn't guarantee reproducibility. Since the tests don't currently use disorderfs, there could be more issues lurking due to filesystem ordering issues. https://r13y.com/ https://r13y.com/
  - Foxboron 4y ago
    Lets not forget that simply using any given build system doesn't magically solve all the issues. If a `make install` invocation gzips manpages instead of the package manager or distro hooks, then you won't have reproducible manpages unless the upstream uses the `-n` switch as well. This is just an example of many small and obscure issues that hide inn different upstream projects.
lloydatkinson 4y ago
Would help if it explains which packages? NPM? Deb? Rpm?
- pabs3 4y ago
  The page mentions openSUSE, ArchLinux, Debian.
ISL 4y ago
The search-engine is case-sensitive.

80 points by pabs3 4y ago

stoplying1 4y ago

(doesn't list or use the package manager most likely to already be reproducibile)

yjftsjthsd-h 4y ago
... Huh. Yeah, that's a really short list of distros, and omitting nix and guix seems like a weird thing in this space.
- pabs3 4y ago
  Seems Nix/Guix are tracked outside of the Reproducible Builds infrastructure, but the site only lists those within that infra. https://reproducible-builds.org/citests/ https://reproducible-builds.org/citests/ https://r13y.com/ https://r13y.com/ https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/package-reproducibility https://data.guix.gnu.org/repository/1/branch/master/latest-...
Foxboron 4y ago
Which package manager would be more reproducible than pacman and Arch? We hover around 85% and 89% reproducible packages across all our packages and no distro, to my knowledge, is close to anything like this. https://reproducible.archlinux.org/ https://reproducible.archlinux.org/
- kaba0 4y ago
  Nix?
  - Foxboron 4y ago
    It's not. It's marketing. The focus of Nix is reproducability in the same wain as docker: Same dependencies for the same behavior. They can't guarantee bit-for-bit reproducability any more then your average distribution. Arch Linux has a rebuilder setup and you can safely reproduce between 85% and 89% of the repository packages. Nix has never done such a thing, and would be far away from any comparable numbers.
    - kaba0 4y ago
      This is literally linked in a reply to you: https://r13y.com/ https://r13y.com/ Nix is 99.77% reproducible, so come on. It’s not marketing, nix just properly solves the problem for the first time ever. Guix is another example, it is more into “bootstrappability” though, so packages can be built from source all the way down to a few kilobyte C-subset compiler written in assembly. Very cool!
      - Foxboron 4y ago
        >Nix is 99.77% reproducible, so come on. It’s not marketing, nix just properly solves the problem for the first time ever. You haven't read r13y properly. >1733 out of 1737 (99.77%) paths in the nixos.iso_minimal.x86_64-linux installation image are reproducible! 600 packages becomes 1737 paths which are the inputs used to generate the image. Several of the compared paths are just static files which doesn't change, so the "path to package" comparison doesn't work either. EDIT: An example set of stuff it attempts to compare can be found here: https://paste.xinu.at/TmVM0y5FBDWX/ https://paste.xinu.at/TmVM0y5FBDWX/ Taken from the build here: https://buildkite.com/grahamc/r13y-dot-com/builds/1149#01813950-9819-49f6-842f-b8e0aa325e1b https://buildkite.com/grahamc/r13y-dot-com/builds/1149#01813... They have not solved the problem.
        kaba0 4y ago
        It’s almost as if no matter how reproducible your dependencies are, if the given build process/tool wants to play dirty it will. I really don’t see how arch fairs any better here, care to elaborate? While it is a different problem, my litmus test for package managers is installing gnome a (the whole group) and then seeing whether I can transition to kde or vice versa without a bunch of dependencies lingering around forever. Pacman, apt, etc. all fail terribly here.
        Foxboron 4y ago
        > I really don’t see how arch fairs any better here, care to elaborate? Arch has spent a considerably amount of time getting to the figure it has and is in the same position as NixOS to maintain reproducible builds. The claim that NixOS solves this, or is uniquely dispositioned to solve this, is the false. That is what I'm effectively trying to point out here. >While it is a different problem, my litmus test for package managers is installing gnome a (the whole group) and then seeing whether I can transition to kde or vice versa without a bunch of dependencies lingering around forever. Pacman, apt, etc. all fail terribly here. That is not what Reproducible Builds is about.

pooper 4y ago

The "weather" icons sunny vs raining vs clock is a blast from the (Jenkins?) past :) For example: https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/ffmpeg.html https://tests.reproducible-builds.org/debian/rb-pkg/unstable...

pabs3 4y ago
The Reproducible Builds test infra indeed runs on Jenkins, specifically the jenkins.debian.net service.

stopdropnhotpot 4y ago

encryptluks2 4y ago

I'm using a hook for pacman on Arch to verify reproducible builds but one issue I've found is it slows down package management substantially. Has anyone else encountered this and is there a solution for verification without slowing down the package manager?

aew4ytasghe5 4y ago
What are you describing here, compile time for a local verification? If so, how would you imagine that could be sped up? - If the distro is working on reproducibility, they should have a dashboard somewhere with reproducibility status. For arch that would be https://reproducible.archlinux.org/ https://reproducible.archlinux.org/ It would be done in their CI and there should be no reason for you do also manually verify this, unless you are a very targeted individual and you think its reasonable that arch linux infrastructure is hacked to target you. Basically, either you trust your distro or you don't. If you are running local verifications of reproducibility, I am guessing you are not. So consider switching to a distro that you do trust.
- encryptluks2 4y ago
  No, not compile time... just verifying binary transparency: https://github.com/kpcyrd/pacman-bintrans https://github.com/kpcyrd/pacman-bintrans

defrost 4y ago

I'm loving the related diffoscope utility (new to me) [1] https://diffoscope.org/ https://diffoscope.org/ It did a fantastic job of highlighting the few changes in a tree of XML "one liner" files (each file one ~ 25 Kb text line).

pabs3 4y ago
BTW, the latest version now checks if two XML files are identical except for ordering differences.
- defrost 4y ago
  Yep - that was the killer feature that sold me! :-)

freeqaz 4y ago

One of my biggest pet peeves with NPM is that the packages aren't reproducible. Even if somebody links a package to their Git repo, who knows what commit correlates to a given release? Even if somebody pushes a "tag" for the release, there is still no guarantee that it's actually the same code. (especially if there is any type of build step.) I'm not 100% certain, but I believe this is an area where Golang is doing better (code references a Git repo directly). Are there any other languages that focus on "reproducible builds" for packages? (Ideally via Git) Edit: I just want to add that this is a major frustration for me with Java too. Everything on Maven barely has any metadata to go back and audit the source code!

vbezhenar 4y ago
What's wrong with maven? You have version in pom.xml, you can retrieve corresponding git tag and audit it.
- chriswarbo 4y ago
  Version numbers are just a form of documentation, giving the developer's best guess of whether the API broke or not. In particular, Maven's use of groupId:artifactId:version is inadequate as an identifier, since many artifacts can have this same "identifier", e.g. - Making changes without updating the version number (bad practice, but easily done) - Building with different versions of dependencies. Maven allows version ranges, and resolves them based on the contents of third-party servers, so this can change over time even if we're building from the same git commit. - Building with different property settings, e.g. due to a different profile getting 'activated' due to some ambient aspect of the system (OS, presence/absence of some file path, env vars, etc.). - Building with different versions of Maven, Java, etc. - The contents of Maven's cache also affects builds. If a file path exists in the cache, Maven will use it regardless of its contents. Also, undeclared dependencies can sometimes be picked up from the cache, which would otherwise fail if they weren't in the cache. Maven also uses that scheme as a location for fetching artifacts, which is not only inadequate but also insecure (we can put arbitrary code at a certain path and Maven will happily fetch it). Maven does perform SHA512 verification, but those hashes are fetched from the same location as the artifacts, so subverting one allows subverting both. Also, in case you weren't aware, git tags are just a convenience. Commit IDs are SHA1 hashes, so they're unforgeable and tamper-proof; whereas git tags can be deleted and recreated to point at anything. FYI at my workplace we avoid some of these issues by: - Performing all builds using Nix, which uses a sandbox with versions of Maven, JVM, etc. specified with their SHA256 hashes. This avoids differences due to tooling, profiles, env vars, caches, etc. - All builds are performed without network access. This prevents Maven resolving version numbers differently. - The SHA256 of the build environment (including all tooling, env vars, properties, source code, etc.) gets appended to the version number, which distinguishes these different artifacts from each other. - Our projects specify the expected hash of their Maven dependencies (essentially like a lock file), which avoids having to trust third-parties for the hashes (a trust-on-first-use policy).

zuzuleinen 4y ago

Using Nix[0] should solve the reproducibility of a package. We use its package manager for bob[1] to achieve reproducible builds for projects. [0] https://nixos.org/guides/how-nix-works.html https://nixos.org/guides/how-nix-works.html [1] https://bob.build/ https://bob.build/

pabs3 4y ago
Unfortunately, not 100% of Nix packages are reproducible, so using it doesn't guarantee reproducibility. Since the tests don't currently use disorderfs, there could be more issues lurking due to filesystem ordering issues. https://r13y.com/ https://r13y.com/
- Foxboron 4y ago
  Lets not forget that simply using any given build system doesn't magically solve all the issues. If a `make install` invocation gzips manpages instead of the package manager or distro hooks, then you won't have reproducible manpages unless the upstream uses the `-n` switch as well. This is just an example of many small and obscure issues that hide inn different upstream projects.

lloydatkinson 4y ago

Would help if it explains which packages? NPM? Deb? Rpm?

pabs3 4y ago
The page mentions openSUSE, ArchLinux, Debian.

ISL 4y ago

The search-engine is case-sensitive.