5 ms·
It's possible but tediuos to run Postgres standalone. I packaged Postgres up for our Bazel build which requires a hermetic environment. The hard parts were: 1.
by sa46 5y ago
It's possible but tediuos to run Postgres standalone. I packaged Postgres up for our Bazel build which requires a hermetic environment. The hard parts were:
1. Postgres doesn't really didn't want to be statically compiled, according to [1].
2. Messing with LD_LIBRARY_PATH to point to an embedded glibc and openssl when running the heremetic postgres binaries.
3. Postgres will absolutely refuse to run the server process as root. That's sensible for a production environment but a pain in CI because it would require setting up another user on machines. I patched away the problem to allow Postgres to run as root.
On the bright side, with some minor tweaking, Postgres can start a fresh database in 300 ms which is plenty fast for tests and you avoid spinning up Docker containers. I ended up using template databases to only spin up the database once per test suite. Each tests then copies the database template for each test in the suite which reduced the setup time per test to ~20 ms.
Instead of using rules_foreign_cc to build Postgres from source with Bazel, I ended up building Postgres outside of Bazel with Docker and zipping it up by target platform (linux|darwin)_(arm64|amd64).
https://www.postgresql.org/message-id/4E0DE1B6.6070600@postnewspapers.com.au https://www.postgresql.org/message-id/4E0DE1B6.6070600@postn...
- mleonard 5y agoVery interesting. Is any of this open source? I'd love to take a look! I'm guessing it's not open source - as I couldn't find it on your Github account or your company's account. If that's the case, would you be able to create a quick gist with a copy/paste of any of the code you can share? If you have the time I'd appreciate it! Related links for anyone interested: - dataform uses bazel for ci tests. They builds redis from source, but run postgres as a container. See here: https://www.reddit.com/r/bazel/comments/kcmbwb/how_to_run_services_for_integration_tests_with/gfsmq6a?utm_source=share&utm_medium=web2x&context=3 https://www.reddit.com/r/bazel/comments/kcmbwb/how_to_run_se...
- sa46 5y agoSure, here's sketch of how it works: https://github.com/jschaf/bazel-postgres-sketch https://github.com/jschaf/bazel-postgres-sketch
- mleonard 5y agoAmazing thanks so much!
- jvolkman 5y agoWe pull Postgres from Nix using rules_nixpkgs for our Bazel tests. It works pretty well. I wrote some additional infrastructure that makes it easy to test against different versions of postgres (11, 12, 13 - all available from Nix) using simple Bazel command line options.
- anarazel 5y ago> 1. Postgres doesn't really didn't want to be statically compiled, according to [1]. Right. There's a lot of features that'll flat out not work if you somehow compile it statically. I don't think it's a useful thing to run tests with a version of postgres built like that. Nor does it really solve anything - postgres will still require a bunch of on-disk files. You shouldn't need have to do so anyway - you can relocate the postgres installation itself when compiled normally. Binaries like postgres will try to find the files they need relative to their own location if not found at the builtin location. Of course, libraries that PG binaries dynamically link to need to be somewhere in the library path. If you need to adjust that you can do it by adding rpaths to the binaries/libraries to additional locations, so you don't need to adjust LD_LIBRARY_PATH.
- sa46 5y agoI ended up skipping static compilation for the reasons you outlined (not necessary). I packaged everything up in a directory and used LD_LIBRARY_PATH. I linked to a sketch on GitHub in a sibling comment.