4 ms·

I'm not a fan of the "Reproducible tarballs" section, because it's explicitly about pre-processing the source code with autotools, instead of distributing a pur

by kpcyrd 3y ago

I'm not a fan of the "Reproducible tarballs" section, because it's explicitly about pre-processing the source code with autotools, instead of distributing a pure, unaltered git snapshot (which `git archive` can already generate in a deterministic way). The section following then mentions signing the pre-processed source code, which I think is the wrong approach. It makes a difficult situation because of how strongly some people encourage signed source code, yet I think autotools is part of the build process and should run on the build server (and double checked by reproducible builds). If people pre-process the .orig.tar.xz they upload to Debian, this pre-processing won't be covered by reproducible builds because it happens undocumented. The patch for "reproducible tarballs" is quite involved[0] and has rookie mistakes like "pin a specific container image using `@sha256:...` syntax, but then invoke `apt-get update` and `apt-get install` to install whatever Debian ships at that time". [0]: https://github.com/curl/curl/pull/13250/files https://github.com/curl/curl/pull/13250/files

rwmj 3y ago
Is git archive deterministic? I think github has trouble here. (No argument that is ought to be deterministic.)
- lrvick 3y ago
  In my experience, yes. Provided it is done with a known git binary etc. Best to containerize workflows like this with hash-locked deps so they can be easily verified by others even far in the future with any OCI compatible toolchain.
- hobofan 3y ago
  The GitHub implementation of git archive does it's best to be deterministic. Some reproducible build systems like e.g. Bazel heavily rely on that. GitHub had a bug early last year[0] that broke that determinism and it caused a huge uproar. So through a mixture of promises to individual projects and just so many projects relying on it, GitHub's git archive has been ossified into being deterministic (unless they want to lose a lot of goodwill among developers). [0]: https://github.com/orgs/community/discussions/45830 https://github.com/orgs/community/discussions/45830
sloowm 3y ago
Maybe you could get involved by pointing out the mistake and proposing the alternative? I imagine that downstream can't easily switch to another distribution method without notice.
throwiforgtnlzy 3y ago
Autotools is slow, ugly, and horrible. There are many better and simpler alternatives that exist now. Source releases shouldn't be pre-processed in any way. They should build whatever they need to on-the-fly as part of the build. Other project anti-patterns: 0. Maintaining creaky, fragile, old forks of other projects as dependencies. 1. Git submodules. 2. Too many dependencies. 3. Extremely large code bases that don't prune poor patterns, over-engineering, and dead features. 4. Lots of features. OpenSSL, I'm looking at you. 5. Not signing releases with a passphrase split between multiple developers to requires 2-person keying. Package management/distro/builder/packager anti-patterns: 6. Patching the hell out of a project without pushing fixes upstream. 7. Inability or failure to source upstream from multiple independent sources, compare them, and verify chain/web of trust using cryptographic signatures. 8. Not following reproducible build guidelines. 9. Not using build caching like sccache. 10. Not building from reproducible sources periodically on the client-side to verify binaries are identical to those built by others. 11. Dependency hell of traditional (non-nix) packages importing zillions of packages.
lrvick 3y ago
Made an alternative PR for reproducible tarballs that should work years from now. https://github.com/curl/curl/pull/13338 https://github.com/curl/curl/pull/13338

by kpcyrd 3y ago

rwmj 3y ago

Is git archive deterministic? I think github has trouble here. (No argument that is ought to be deterministic.)

lrvick 3y ago
In my experience, yes. Provided it is done with a known git binary etc. Best to containerize workflows like this with hash-locked deps so they can be easily verified by others even far in the future with any OCI compatible toolchain.
hobofan 3y ago
The GitHub implementation of git archive does it's best to be deterministic. Some reproducible build systems like e.g. Bazel heavily rely on that. GitHub had a bug early last year[0] that broke that determinism and it caused a huge uproar. So through a mixture of promises to individual projects and just so many projects relying on it, GitHub's git archive has been ossified into being deterministic (unless they want to lose a lot of goodwill among developers). [0]: https://github.com/orgs/community/discussions/45830 https://github.com/orgs/community/discussions/45830

sloowm 3y ago

Maybe you could get involved by pointing out the mistake and proposing the alternative? I imagine that downstream can't easily switch to another distribution method without notice.

throwiforgtnlzy 3y ago

Autotools is slow, ugly, and horrible. There are many better and simpler alternatives that exist now. Source releases shouldn't be pre-processed in any way. They should build whatever they need to on-the-fly as part of the build. Other project anti-patterns: 0. Maintaining creaky, fragile, old forks of other projects as dependencies. 1. Git submodules. 2. Too many dependencies. 3. Extremely large code bases that don't prune poor patterns, over-engineering, and dead features. 4. Lots of features. OpenSSL, I'm looking at you. 5. Not signing releases with a passphrase split between multiple developers to requires 2-person keying. Package management/distro/builder/packager anti-patterns: 6. Patching the hell out of a project without pushing fixes upstream. 7. Inability or failure to source upstream from multiple independent sources, compare them, and verify chain/web of trust using cryptographic signatures. 8. Not following reproducible build guidelines. 9. Not using build caching like sccache. 10. Not building from reproducible sources periodically on the client-side to verify binaries are identical to those built by others. 11. Dependency hell of traditional (non-nix) packages importing zillions of packages.

lrvick 3y ago

Made an alternative PR for reproducible tarballs that should work years from now. https://github.com/curl/curl/pull/13338 https://github.com/curl/curl/pull/13338