6 ms·

The new PostgreSQL 17 make dist

206 points by ingve 2y ago

mgaunard 2y ago
tl;dr they're moving to using "git archive" to ensure what's in the tarball is what's under git.
carderne 2y ago
Does this make the downstream packagers’ jobs harder — they must now presumably have Perl, Bison, Flex and DocBook installed on the packaging machines?
- b112 2y ago
  Everything but docbook seems pretty standard.
- mhio 2y ago
  Debian already had them https://salsa.debian.org/postgresql/postgresql/-/blob/16/debian/control#L13-20 https://salsa.debian.org/postgresql/postgresql/-/blob/16/deb... https://salsa.debian.org/postgresql/postgresql/-/compare/16...17#58ef006ab62b83b4bec5d81fe5b32c3b4c2d1cc2 https://salsa.debian.org/postgresql/postgresql/-/compare/16....
- kpcyrd 2y ago
  The build server now needs the actual build dependencies instead of relying on pre-built intermediate build artifacts. This is a good thing, and should be expected from anything that claims to "build from source".
  - carderne 2y ago
    I know it’s a good thing. I’m just curious to what degree packagers see this as extra work. Seems like probably not.
    - vsl 2y ago
      It's going to be a trivial amount of extra work. Packaging build systems (well, the mainstream ones: Debian, RPMs, ebuilds etc.) have the concept of runtime and build-time dependencies, so the maintainers will just need to add a few more packages to the build deps list.
    - EuAndreh 2y ago
      IIRC Debian has a policy on preferring to generated files themselves. I couldn't find the link to it right now, though.
      - EuAndreh 2y ago
        s/generated/generate/
      - pabs3 2y ago
        Some info on here: https://wiki.debian.org/AutoGeneratedFiles https://wiki.debian.org/AutoGeneratedFiles
        EuAndreh 2y ago
        Yep, that says not to include generated files in VCS. I've also seen the bison and flex output included in VCS, and the same guidelines apply.
    - bluGill 2y ago
      Many distributions have a policy of not using generated files, and more are implementing it after the xz hack. Most package systems have (not just Debian) have the concept of build and deploy dependencies. It is of course more work, and package systems tend to not be well documented and so it is hard to figure out, but it is possible: if you ask the experts they will help
- chubot 2y ago
  Hm the article was pretty unclear to me, and I read several times 1) Do they commit the generated flex/bison to git now? So tarballs match git 2) Or do they now leave it up to the end user to generate flex/bison? And run their custom Perl scripts, etc.
  - sjamaan 2y ago
    As I understood it, they used to generate the flex/bison stuff because they relied on very specific versions of those. But now those versions are commonly available, they rely on the user to generate these.
    - gpvos 2y ago
      I understood they just need to be recent enough.
  - gpvos 2y ago
    2.
- XorNot 2y ago
  This doesn't seem like a problem though? As long as it's open source with packages available, that's just a CI process. At some point those machines are building those packages anyway, so the content is also already cached.
steeleduncan 2y ago
Nix and Guix have their issues, but it is hard to read something like this and not wonder why you would migrate to them when facing issues like this There is a learning curve for either Nix or Guix that puts many off. However its not that steep, certainly it is many orders of magnitude easier than maintaining PostgreSQL, and once you are over that you no longer need to do things like keeping a dedicated clean machine just to pack a tarball. Write the derivation and anyone, anywhere, on any machine can generate the exact same tarball with a one liner The barrier caused by the initial steps of learning Nix/Guix is a shame because once you are over it, it is difficult to see why software is built any other way (the same may apply to bazel, but i have no experience with that).
- hddqsb 2y ago
  Docker & co. also let you create a clean build environment (to a lesser extent), and I find them less intrusive than Nix / Guix.
  - Nullabillity 2y ago
    Docker isn't reproducible. The one thing it can give you is a consistent set of mystery meat binaries, but that's an even worse starting point than the old problem of mystery meat source code.
    - ffsm8 2y ago
      Docker images can be reproducible. They just aren't by default (because they include a timestamp) and you need to jump through multiple hoops to get them there, consistently. (And things like "apk add" or "apt install" can't be used unless you're installing pinned versions)
      - sgarland 2y ago
        Why would you not be pinning versions in a Dockerfile? The entire point is “if it works on my machine, it works on yours,” and that goes out the window if you can’t be assured that every program in the release is at the same version you had.
hlandau 2y ago
Personally I've always considered it bad hygiene to commit generated outputs, but this article notes that this takes on a new significance in the light of supply chain security concerns. Good changes from PostgreSQL here. Generated output, vendored source trees, etc. aren't, or can't be, meaningfully audited as part of a code review process, so they're basically merged without real audit or verification. My personal preference is never to include generated output in a repository or tarball, including e.g. autoconf/automake scripts. This is directly contrary to the advice of the autotools documentation, which wants people to ship these unauditably gargantuan and obtuse generated scripts as part of tarballs... an approach which created an ideal space for things like the XZ backdoor.
- nrabulinski 2y ago
  My take is that they should always be committed, but never generated by the dev, instead generated and pushed when necessary by CI. The problem with generating those files yourself is that, in many cases, it makes the output nondeterministic and nonreproducible. In the ideal world those tools would just generate those files deterministically, but until then for me committing them from CI is an acceptable stopgap
  - koolba 2y ago
    My preference is to do both. Have them generated by a dev, committed, and also generated in CI. The latter gets compared with the checked in contents to ensure the results match the expected value. This speeds up CI (the generation path can be done in parallel) and most local development. The one catch is that it relies on mostly trusting whoever has a commit bit. But if you don’t have that and any part of the build involves scripts that are part of the repo itself, then you’ve already lost.
    - giancarlostoro 2y ago
      > The one catch is that it relies on mostly trusting whoever has a commit bit. Would the comparison not show that the person you're trusting goofed or is being malicious?
alexvitkov 2y ago
Flex and Bison are simple enough tools (probably, you never know with GNU stuff) that you should be able to just vendor & compile them as part of the build process if you actually care about reproducible builds.
- EuAndreh 2y ago
  yacc and lex, their description and some implementations are simple. Flex and certainly Bison are not.
gpvos 2y ago
> Currently, the Git version used to produce the release tarballs (on the above-mentioned “clean” box) is too old to create reproducible .tar.gz tarballs, but it will create reproducible .tar.bz2 tarballs. What is different about gzip and bzip2 that causes this?
- stonemetal12 2y ago
  Gzip just isn't specified well enough to be fully reproduceable. Gzip is technically only a file format specification, with multiple compression algorithms allowed. Also Gzip includes metadata about the compression like a timestamp for when the file was compressed, and bits for which OS was in use when the file was compressed, etc. so the out put is never 100% the same, though it ought to be easy to work around that part.
- o11c 2y ago
  Presumably, the internal gzip lacks support for -n. This could be fixed regardless of git version by calling external gzip after generating a plain tarball (or even decompressing the bzipped tarball). Hmm, I wonder which approach would actually be fastest. Does cache contention break the obvious use of `tee(1)`?
- petereisentraut 2y ago
  Git 2.38.0 is the version where git archive uses an internal gzip implementation instead of calling the actual external gzip. This internal implementation has two improvements for this purpose: First, it doesn't store the timestamp. You could also get that with gzip -n. (But the old git archive didn't do that, so you have to run the gzip as a separate step after git archive.) Second, it stores the platform identification bits as "UNIX" on all platforms, so the output is identical on all platforms. There is no gzip command-line option for that, unfortunately.
  - gpvos 2y ago
    Ah right. My suggestion would have been to use a small Perl or Python script or something, assuming there would be a zip library that would let you set the proper flags.
iamcreasy 2y ago
> Some packagers have policies that everything needs to be built from source, so they’d just delete and rebuild the prebuilt files anyway. What packages are they referring?
cryptonector 2y ago
Historically in autoconf codebases (which PostgreSQL is) `make dist` is done: a) after `./configure` (so you have a Makefile, naturally), which is b) after `autoreconf -fi` (so you have a `./configure`). This allows the dist archive to contain the outputs of `autoreconf -fi` so that users need not have autoconf installed and they can just run `./configure`. Switching to `git archive` is fine, and you can add files to that, but https://github.com/postgres/postgres/blob/master/GNUmakefile.in#L107 https://github.com/postgres/postgres/blob/master/GNUmakefile... doesn't. So, I guess users now _have to_ run `autoreconf -fi`? No, because those are now committed in the source tree (https://github.com/postgres/postgres/blob/master/configure https://github.com/postgres/postgres/blob/master/configure).
- petereisentraut 2y ago
  The configure generated from configure.ac has always been checked into Git for PostgreSQL. So with either the old or the new make dist approach, the configure in the tarball matches the one checked into the source code repository. So this is outside of what this article is discussing.
  - cryptonector 2y ago
    Got it. I should have checked first.
klysm 2y ago
The death of autoconf could be one of the biggest wins for the software world

206 points by ingve 2y ago

tl;dr they're moving to using "git archive" to ensure what's in the tarball is what's under git.

Does this make the downstream packagers’ jobs harder — they must now presumably have Perl, Bison, Flex and DocBook installed on the packaging machines?

b112 2y ago
Everything but docbook seems pretty standard.
mhio 2y ago
Debian already had them https://salsa.debian.org/postgresql/postgresql/-/blob/16/debian/control#L13-20 https://salsa.debian.org/postgresql/postgresql/-/blob/16/deb... https://salsa.debian.org/postgresql/postgresql/-/compare/16...17#58ef006ab62b83b4bec5d81fe5b32c3b4c2d1cc2 https://salsa.debian.org/postgresql/postgresql/-/compare/16....
kpcyrd 2y ago
The build server now needs the actual build dependencies instead of relying on pre-built intermediate build artifacts. This is a good thing, and should be expected from anything that claims to "build from source".
- carderne 2y ago
  I know it’s a good thing. I’m just curious to what degree packagers see this as extra work. Seems like probably not.
  - vsl 2y ago
    It's going to be a trivial amount of extra work. Packaging build systems (well, the mainstream ones: Debian, RPMs, ebuilds etc.) have the concept of runtime and build-time dependencies, so the maintainers will just need to add a few more packages to the build deps list.
  - EuAndreh 2y ago
    IIRC Debian has a policy on preferring to generated files themselves. I couldn't find the link to it right now, though.
    - EuAndreh 2y ago
      s/generated/generate/
    - pabs3 2y ago
      Some info on here: https://wiki.debian.org/AutoGeneratedFiles https://wiki.debian.org/AutoGeneratedFiles
      - EuAndreh 2y ago
        Yep, that says not to include generated files in VCS. I've also seen the bison and flex output included in VCS, and the same guidelines apply.
  - bluGill 2y ago
    Many distributions have a policy of not using generated files, and more are implementing it after the xz hack. Most package systems have (not just Debian) have the concept of build and deploy dependencies. It is of course more work, and package systems tend to not be well documented and so it is hard to figure out, but it is possible: if you ask the experts they will help
chubot 2y ago
Hm the article was pretty unclear to me, and I read several times 1) Do they commit the generated flex/bison to git now? So tarballs match git 2) Or do they now leave it up to the end user to generate flex/bison? And run their custom Perl scripts, etc.
- sjamaan 2y ago
  As I understood it, they used to generate the flex/bison stuff because they relied on very specific versions of those. But now those versions are commonly available, they rely on the user to generate these.
  - gpvos 2y ago
    I understood they just need to be recent enough.
- gpvos 2y ago
  2.
XorNot 2y ago
This doesn't seem like a problem though? As long as it's open source with packages available, that's just a CI process. At some point those machines are building those packages anyway, so the content is also already cached.

steeleduncan 2y ago

Nix and Guix have their issues, but it is hard to read something like this and not wonder why you would migrate to them when facing issues like this There is a learning curve for either Nix or Guix that puts many off. However its not that steep, certainly it is many orders of magnitude easier than maintaining PostgreSQL, and once you are over that you no longer need to do things like keeping a dedicated clean machine just to pack a tarball. Write the derivation and anyone, anywhere, on any machine can generate the exact same tarball with a one liner The barrier caused by the initial steps of learning Nix/Guix is a shame because once you are over it, it is difficult to see why software is built any other way (the same may apply to bazel, but i have no experience with that).

hddqsb 2y ago
Docker & co. also let you create a clean build environment (to a lesser extent), and I find them less intrusive than Nix / Guix.
- Nullabillity 2y ago
  Docker isn't reproducible. The one thing it can give you is a consistent set of mystery meat binaries, but that's an even worse starting point than the old problem of mystery meat source code.
  - ffsm8 2y ago
    Docker images can be reproducible. They just aren't by default (because they include a timestamp) and you need to jump through multiple hoops to get them there, consistently. (And things like "apk add" or "apt install" can't be used unless you're installing pinned versions)
    - sgarland 2y ago
      Why would you not be pinning versions in a Dockerfile? The entire point is “if it works on my machine, it works on yours,” and that goes out the window if you can’t be assured that every program in the release is at the same version you had.

hlandau 2y ago

Personally I've always considered it bad hygiene to commit generated outputs, but this article notes that this takes on a new significance in the light of supply chain security concerns. Good changes from PostgreSQL here. Generated output, vendored source trees, etc. aren't, or can't be, meaningfully audited as part of a code review process, so they're basically merged without real audit or verification. My personal preference is never to include generated output in a repository or tarball, including e.g. autoconf/automake scripts. This is directly contrary to the advice of the autotools documentation, which wants people to ship these unauditably gargantuan and obtuse generated scripts as part of tarballs... an approach which created an ideal space for things like the XZ backdoor.

nrabulinski 2y ago
My take is that they should always be committed, but never generated by the dev, instead generated and pushed when necessary by CI. The problem with generating those files yourself is that, in many cases, it makes the output nondeterministic and nonreproducible. In the ideal world those tools would just generate those files deterministically, but until then for me committing them from CI is an acceptable stopgap
- koolba 2y ago
  My preference is to do both. Have them generated by a dev, committed, and also generated in CI. The latter gets compared with the checked in contents to ensure the results match the expected value. This speeds up CI (the generation path can be done in parallel) and most local development. The one catch is that it relies on mostly trusting whoever has a commit bit. But if you don’t have that and any part of the build involves scripts that are part of the repo itself, then you’ve already lost.
  - giancarlostoro 2y ago
    > The one catch is that it relies on mostly trusting whoever has a commit bit. Would the comparison not show that the person you're trusting goofed or is being malicious?

alexvitkov 2y ago

Flex and Bison are simple enough tools (probably, you never know with GNU stuff) that you should be able to just vendor & compile them as part of the build process if you actually care about reproducible builds.

EuAndreh 2y ago
yacc and lex, their description and some implementations are simple. Flex and certainly Bison are not.

gpvos 2y ago

> Currently, the Git version used to produce the release tarballs (on the above-mentioned “clean” box) is too old to create reproducible .tar.gz tarballs, but it will create reproducible .tar.bz2 tarballs. What is different about gzip and bzip2 that causes this?

stonemetal12 2y ago
Gzip just isn't specified well enough to be fully reproduceable. Gzip is technically only a file format specification, with multiple compression algorithms allowed. Also Gzip includes metadata about the compression like a timestamp for when the file was compressed, and bits for which OS was in use when the file was compressed, etc. so the out put is never 100% the same, though it ought to be easy to work around that part.
o11c 2y ago
Presumably, the internal gzip lacks support for -n. This could be fixed regardless of git version by calling external gzip after generating a plain tarball (or even decompressing the bzipped tarball). Hmm, I wonder which approach would actually be fastest. Does cache contention break the obvious use of `tee(1)`?
petereisentraut 2y ago
Git 2.38.0 is the version where git archive uses an internal gzip implementation instead of calling the actual external gzip. This internal implementation has two improvements for this purpose: First, it doesn't store the timestamp. You could also get that with gzip -n. (But the old git archive didn't do that, so you have to run the gzip as a separate step after git archive.) Second, it stores the platform identification bits as "UNIX" on all platforms, so the output is identical on all platforms. There is no gzip command-line option for that, unfortunately.
- gpvos 2y ago
  Ah right. My suggestion would have been to use a small Perl or Python script or something, assuming there would be a zip library that would let you set the proper flags.

iamcreasy 2y ago

> Some packagers have policies that everything needs to be built from source, so they’d just delete and rebuild the prebuilt files anyway. What packages are they referring?

cryptonector 2y ago

Historically in autoconf codebases (which PostgreSQL is) `make dist` is done: a) after `./configure` (so you have a Makefile, naturally), which is b) after `autoreconf -fi` (so you have a `./configure`). This allows the dist archive to contain the outputs of `autoreconf -fi` so that users need not have autoconf installed and they can just run `./configure`. Switching to `git archive` is fine, and you can add files to that, but https://github.com/postgres/postgres/blob/master/GNUmakefile.in#L107 https://github.com/postgres/postgres/blob/master/GNUmakefile... doesn't. So, I guess users now _have to_ run `autoreconf -fi`? No, because those are now committed in the source tree (https://github.com/postgres/postgres/blob/master/configure https://github.com/postgres/postgres/blob/master/configure).

petereisentraut 2y ago
The configure generated from configure.ac has always been checked into Git for PostgreSQL. So with either the old or the new make dist approach, the configure in the tarball matches the one checked into the source code repository. So this is outside of what this article is discussing.
- cryptonector 2y ago
  Got it. I should have checked first.

klysm 2y ago

The death of autoconf could be one of the biggest wins for the software world