3 ms·

I must say I'm quite pleased to see how well Go version works. It does only use 1.5x the CPU and (predictably) much more RAM/VRAM, but not a crazy amount either

I must say I'm quite pleased to see how well Go version works. It does only use 1.5x the CPU and (predictably) much more RAM/VRAM, but not a crazy amount either (the expected increase is 2x). Of course you can write a more optimal version in C / C++ / Zig / Rust, but at the same time Go is much easier to write and you don't pay for the convenience with an absurd performance loss like in Python or PHP

__s 4mo ago
indeed, wal-g actually started as a port of wal-e which was Python: https://www.citusdata.com/blog/2017/08/18/introducing-wal-g-faster-restores-for-postgres https://www.citusdata.com/blog/2017/08/18/introducing-wal-g-... wal-g was a much larger improvement over wal-e. we're optimizing the margins here
- sgt 4mo ago
  I'd like to use the one getting the most community support. So too soon to wait for Rust vs Go. Although on paper, Rust is better.
  - __s 4mo ago
    tbf it took 4 years since PG15 support was added for me to fix remote BASE_BACKUP support & wal-g base backups being inconsistent on PG15+ (parameter typo had pg_backup_stop return before wal archived far enough for consistency) https://github.com/wal-g/wal-g/pull/2262 https://github.com/wal-g/wal-g/pull/2262 but yes, this is young project, so fair take
- saisrirampur 4mo ago
  ++ We’re big Go fans, most of PeerDB is written in Go: https://github.com/PeerDB-io/peerdb https://github.com/PeerDB-io/peerdb The importance of optimizing (resource) margins and having predictable memory usage increases significantly in the DBaaS/Postgres world, where your process coexists and competes with other critical workloads. Also, WAL-RUS isn’t rocket science. Postgres already exposes a bunch native constructs for WAL archival, making development fairly straightforward even in Rust. TL;DR: when to choose Rust or Go really depends on the workload and what you are going after.
  - keynha 4mo ago
    That coexistence is also why the GC-free rewrite helps more than the speed numbers suggest. An archiver is allocation-light until it hits a compression burst, then the Go heap can spike toward 2x live right when Postgres wants that memory. GOMEMLIMIT caps the spike but pays in GC CPU during exactly those bursts, so on a small instance, you are trading OOM risk for throughput. Rust removes that dial.
pjmlp 4mo ago
In many cases it could even be more optimised, but somehow using value types is a lost art. Granted Go could make it much more easier to understand when they escape the stack into the heap, as many of their design decisions, simple and easy aren't the same.

__s 4mo ago

indeed, wal-g actually started as a port of wal-e which was Python: https://www.citusdata.com/blog/2017/08/18/introducing-wal-g-faster-restores-for-postgres https://www.citusdata.com/blog/2017/08/18/introducing-wal-g-... wal-g was a much larger improvement over wal-e. we're optimizing the margins here

sgt 4mo ago
I'd like to use the one getting the most community support. So too soon to wait for Rust vs Go. Although on paper, Rust is better.
- __s 4mo ago
  tbf it took 4 years since PG15 support was added for me to fix remote BASE_BACKUP support & wal-g base backups being inconsistent on PG15+ (parameter typo had pg_backup_stop return before wal archived far enough for consistency) https://github.com/wal-g/wal-g/pull/2262 https://github.com/wal-g/wal-g/pull/2262 but yes, this is young project, so fair take
saisrirampur 4mo ago
++ We’re big Go fans, most of PeerDB is written in Go: https://github.com/PeerDB-io/peerdb https://github.com/PeerDB-io/peerdb The importance of optimizing (resource) margins and having predictable memory usage increases significantly in the DBaaS/Postgres world, where your process coexists and competes with other critical workloads. Also, WAL-RUS isn’t rocket science. Postgres already exposes a bunch native constructs for WAL archival, making development fairly straightforward even in Rust. TL;DR: when to choose Rust or Go really depends on the workload and what you are going after.
- keynha 4mo ago
  That coexistence is also why the GC-free rewrite helps more than the speed numbers suggest. An archiver is allocation-light until it hits a compression burst, then the Go heap can spike toward 2x live right when Postgres wants that memory. GOMEMLIMIT caps the spike but pays in GC CPU during exactly those bursts, so on a small instance, you are trading OOM risk for throughput. Rust removes that dial.

pjmlp 4mo ago

In many cases it could even be more optimised, but somehow using value types is a lost art. Granted Go could make it much more easier to understand when they escape the stack into the heap, as many of their design decisions, simple and easy aren't the same.