7 ms·

Running an independent Arch Linux rebuilder

127 points by kpcyrd 6y ago

eindiran 6y ago
Based on my reading of this Arch wiki page[0], it looks like this doesn't impact packages in the AUR. Does anyone who is more familiar with this know if that is true? It appears that it does work for community packages though (see the bottom of this page[1]). [0] https://wiki.archlinux.org/index.php/Rebuilderd https://wiki.archlinux.org/index.php/Rebuilderd [1] https://wiki.archlinux.org/index.php/Rebuilderd#Syncing_packages_to_rebuild https://wiki.archlinux.org/index.php/Rebuilderd#Syncing_pack...
- tomjakubowski 6y ago
  AUR packages are source only so there's no binary package that is distributed to be verified.
  - serf 6y ago
    that's only mostly true. take for example the aur package for MS fonts; https://aur.archlinux.org/packages/ttf-ms-fonts/ https://aur.archlinux.org/packages/ttf-ms-fonts/ all it does is download a ton of microsoft corefont executables to unpack. another example would be the proprietary driver packages, like the nvidia ones.
    - JeremyNT 6y ago
      You're describing what those AUR builds do not what they are. Reproducible builds are for artifacts that are... built. Plenty of AUR recipes download blobs and explode them, but the resulting xz package isn't actually hosted in the AUR. A tool like rebuilderd would only be relevant to the AUR in the case that somebody is publishing generated AUR packages to a repo for others to install without building them directly, and then you'd like to verify those.
    - banachtarski 6y ago
      proprietary drivers from nvidia are provided by the Arch mainline (core) though?
      - serf 6y ago
        right you are, i've been on beta for years and had forgotten. point still stands, aur does distribute executables once in awhile.
        pritambaral 6y ago
        > aur does distribute executables once in awhile. The AUR doesn't distribute those executables; it distributes the instructions for the user (or some software working on behalf of the user) to go fetch the executables themselves. This is an important distinction, because the official Arch repositories _do_ distribute executables, directly to the user.
    - tomjakubowski 6y ago
      This is a question of the artifacts distributed as part of Arch Linux, the tar.xz packages. The idea is to be able to reproducibly build a package from source (where "source", yes, may include binary assets and blobs), and check that the copy you built matches what Arch distributes, byte for byte. This way, you know the packager hasn't introduced unexpected changes. Since end users build AUR packages themselves, there is nothing distributed, and nothing to verify.
    - gerdesj 6y ago
      I think of the AUR packages as more like ebuilds with fewer safety catches 8)
- Foxboron 6y ago
  >Based on my reading of this Arch wiki page[0], it looks like this doesn't impact packages in the AUR. Yes. AUR is excluded from reproducible builds because it only provides build scripts and not actual packages. There is nothing to reproduce. One surely could make a repo and compare results, but it would frankly be a bit futile considering the general package quality there.
usr1106 6y ago
The wiki says: > a large number of builds are not reproducible yet
- Foxboron 6y ago
  Yes. But a staggering amount of packages in the main [core] repository is still fully reproducible by independent parties if tools are run. https://wiki.archlinux.org/index.php/DeveloperWiki:ReproduciblePackages https://wiki.archlinux.org/index.php/DeveloperWiki:Reproduci... It's still very much a work in progress.
elagost 6y ago
This is fantastic to see. Reproducible builds adds yet another layer of trust on top of open source software. The wiki page is also classic Arch Wiki-style, with all the detail one would expect. Every OSS project should strive to be this helpful.
Foxboron 6y ago
I never got around to submitting my blog to HN. But if people are curious about some technical details of the underlying problems of reproducing Arch Linux packages I wrote something a few months ago. https://linderud.dev/blog/reproducible-arch-linux-packages/ https://linderud.dev/blog/reproducible-arch-linux-packages/ EDIT: I did self-discover I did submit the article when I wrote it. Just forgot. Oh well :)
richardwhiuk 6y ago
Debian have done a huge amount of work in this area - https://wiki.debian.org/ReproducibleBuilds https://wiki.debian.org/ReproducibleBuilds
- samueloph 6y ago
  I believe the whole reproducible-builds.org project started at Debian [0], the Core Team is still all made of Debian Developers, more information at: https://reproducible-builds.org/ https://reproducible-builds.org/ Their documentation has some really good pointers at common issues and how to solve them: https://reproducible-builds.org/docs/ https://reproducible-builds.org/docs/ And the diffoscope tool is very useful for debugging. [0] I believe other projects had similar things going on but the Debian one became the main to serve as an umbrella.
  - pabs3 6y ago
    Some more reproducible builds history (it has been around since at least the 1990s): https://wiki.debian.org/ReproducibleBuilds/History https://wiki.debian.org/ReproducibleBuilds/History
- elagost 6y ago
  Makes sense, given Chris Lamb, former Debian Project Leader, is heavily involved. https://reproducible-builds.org/who/ https://reproducible-builds.org/who/
Bnshsysjab 6y ago
I care more about if maintainers actually audit the contents of packages rather than if their builds are reproducible (though the latter still matters)! Not just is there obvious malware, but also are there obvious vulnerabilities, is the person that wrote it of good nature / located in a country where they’re safe from nation state pressure, is there a lot of history behind the app. Obviously this is too much work for any individual and requires a chain of trust. I believe fedora and Ubuntu at the very least audit to some extent but I’ve never seen any doco.
- andr0x 6y ago
  Just curious, are you saying there is obvious malware within the Arch official repositories? Are you referring to instead the user repositories (AUR)? I know Arch has: https://wiki.archlinux.org/index.php/Arch_Security_Team https://wiki.archlinux.org/index.php/Arch_Security_Team but I'd be really interested to hear that the official packages have obvious malware.
  - saghm 6y ago
    I think they intended the phrase as a question; they're proposing that a maintainer should not just ask "is there obvious malware in this package?", but also several other questions (which they list afterwards).
    - Bnshsysjab 6y ago
      This is the correct interpret
- karlding 6y ago
  Isn't this chain of trust essentially what crev [0] tries to do? FWIW Rust implements cargo-crev [1], but I suppose you could extend this to AUR packages with a bit of work. [0] https://github.com/crev-dev/crev/ https://github.com/crev-dev/crev/ [1] https://github.com/crev-dev/cargo-crev https://github.com/crev-dev/cargo-crev
captn3m0 6y ago
Interesting to see https://github.com/kpcyrd/rebuilderd https://github.com/kpcyrd/rebuilderd in Rust. Is this the first Arch project in Rust?
- jelly1 6y ago
  I guess,.. but this FOSDEM we hacked on more Rust based tooling :-) https://github.com/archlinux/rebuilder https://github.com/archlinux/rebuilder https://github.com/archlinux/signstar https://github.com/archlinux/signstar
snvzz 6y ago
This is very good news. I was wondering recently just how to do this, as I wanted to build everything with some different compiler configuration.

127 points by kpcyrd 6y ago

Based on my reading of this Arch wiki page[0], it looks like this doesn't impact packages in the AUR. Does anyone who is more familiar with this know if that is true? It appears that it does work for community packages though (see the bottom of this page[1]). [0] https://wiki.archlinux.org/index.php/Rebuilderd https://wiki.archlinux.org/index.php/Rebuilderd [1] https://wiki.archlinux.org/index.php/Rebuilderd#Syncing_packages_to_rebuild https://wiki.archlinux.org/index.php/Rebuilderd#Syncing_pack...

tomjakubowski 6y ago
AUR packages are source only so there's no binary package that is distributed to be verified.
- serf 6y ago
  that's only mostly true. take for example the aur package for MS fonts; https://aur.archlinux.org/packages/ttf-ms-fonts/ https://aur.archlinux.org/packages/ttf-ms-fonts/ all it does is download a ton of microsoft corefont executables to unpack. another example would be the proprietary driver packages, like the nvidia ones.
  - JeremyNT 6y ago
    You're describing what those AUR builds do not what they are. Reproducible builds are for artifacts that are... built. Plenty of AUR recipes download blobs and explode them, but the resulting xz package isn't actually hosted in the AUR. A tool like rebuilderd would only be relevant to the AUR in the case that somebody is publishing generated AUR packages to a repo for others to install without building them directly, and then you'd like to verify those.
  - banachtarski 6y ago
    proprietary drivers from nvidia are provided by the Arch mainline (core) though?
    - serf 6y ago
      right you are, i've been on beta for years and had forgotten. point still stands, aur does distribute executables once in awhile.
      - pritambaral 6y ago
        > aur does distribute executables once in awhile. The AUR doesn't distribute those executables; it distributes the instructions for the user (or some software working on behalf of the user) to go fetch the executables themselves. This is an important distinction, because the official Arch repositories _do_ distribute executables, directly to the user.
  - tomjakubowski 6y ago
    This is a question of the artifacts distributed as part of Arch Linux, the tar.xz packages. The idea is to be able to reproducibly build a package from source (where "source", yes, may include binary assets and blobs), and check that the copy you built matches what Arch distributes, byte for byte. This way, you know the packager hasn't introduced unexpected changes. Since end users build AUR packages themselves, there is nothing distributed, and nothing to verify.
  - gerdesj 6y ago
    I think of the AUR packages as more like ebuilds with fewer safety catches 8)
Foxboron 6y ago
>Based on my reading of this Arch wiki page[0], it looks like this doesn't impact packages in the AUR. Yes. AUR is excluded from reproducible builds because it only provides build scripts and not actual packages. There is nothing to reproduce. One surely could make a repo and compare results, but it would frankly be a bit futile considering the general package quality there.

usr1106 6y ago

The wiki says: > a large number of builds are not reproducible yet

Foxboron 6y ago
Yes. But a staggering amount of packages in the main [core] repository is still fully reproducible by independent parties if tools are run. https://wiki.archlinux.org/index.php/DeveloperWiki:ReproduciblePackages https://wiki.archlinux.org/index.php/DeveloperWiki:Reproduci... It's still very much a work in progress.

elagost 6y ago

This is fantastic to see. Reproducible builds adds yet another layer of trust on top of open source software. The wiki page is also classic Arch Wiki-style, with all the detail one would expect. Every OSS project should strive to be this helpful.

Foxboron 6y ago

I never got around to submitting my blog to HN. But if people are curious about some technical details of the underlying problems of reproducing Arch Linux packages I wrote something a few months ago. https://linderud.dev/blog/reproducible-arch-linux-packages/ https://linderud.dev/blog/reproducible-arch-linux-packages/ EDIT: I did self-discover I did submit the article when I wrote it. Just forgot. Oh well :)

richardwhiuk 6y ago

Debian have done a huge amount of work in this area - https://wiki.debian.org/ReproducibleBuilds https://wiki.debian.org/ReproducibleBuilds

samueloph 6y ago
I believe the whole reproducible-builds.org project started at Debian [0], the Core Team is still all made of Debian Developers, more information at: https://reproducible-builds.org/ https://reproducible-builds.org/ Their documentation has some really good pointers at common issues and how to solve them: https://reproducible-builds.org/docs/ https://reproducible-builds.org/docs/ And the diffoscope tool is very useful for debugging. [0] I believe other projects had similar things going on but the Debian one became the main to serve as an umbrella.
- pabs3 6y ago
  Some more reproducible builds history (it has been around since at least the 1990s): https://wiki.debian.org/ReproducibleBuilds/History https://wiki.debian.org/ReproducibleBuilds/History
elagost 6y ago
Makes sense, given Chris Lamb, former Debian Project Leader, is heavily involved. https://reproducible-builds.org/who/ https://reproducible-builds.org/who/

Bnshsysjab 6y ago

I care more about if maintainers actually audit the contents of packages rather than if their builds are reproducible (though the latter still matters)! Not just is there obvious malware, but also are there obvious vulnerabilities, is the person that wrote it of good nature / located in a country where they’re safe from nation state pressure, is there a lot of history behind the app. Obviously this is too much work for any individual and requires a chain of trust. I believe fedora and Ubuntu at the very least audit to some extent but I’ve never seen any doco.

andr0x 6y ago
Just curious, are you saying there is obvious malware within the Arch official repositories? Are you referring to instead the user repositories (AUR)? I know Arch has: https://wiki.archlinux.org/index.php/Arch_Security_Team https://wiki.archlinux.org/index.php/Arch_Security_Team but I'd be really interested to hear that the official packages have obvious malware.
- saghm 6y ago
  I think they intended the phrase as a question; they're proposing that a maintainer should not just ask "is there obvious malware in this package?", but also several other questions (which they list afterwards).
  - Bnshsysjab 6y ago
    This is the correct interpret
karlding 6y ago
Isn't this chain of trust essentially what crev [0] tries to do? FWIW Rust implements cargo-crev [1], but I suppose you could extend this to AUR packages with a bit of work. [0] https://github.com/crev-dev/crev/ https://github.com/crev-dev/crev/ [1] https://github.com/crev-dev/cargo-crev https://github.com/crev-dev/cargo-crev

captn3m0 6y ago

Interesting to see https://github.com/kpcyrd/rebuilderd https://github.com/kpcyrd/rebuilderd in Rust. Is this the first Arch project in Rust?

jelly1 6y ago
I guess,.. but this FOSDEM we hacked on more Rust based tooling :-) https://github.com/archlinux/rebuilder https://github.com/archlinux/rebuilder https://github.com/archlinux/signstar https://github.com/archlinux/signstar

snvzz 6y ago

This is very good news. I was wondering recently just how to do this, as I wanted to build everything with some different compiler configuration.