3 ms·

Show HN: pgwire-replication - pure rust client for Postgres CDC

46 points by sacs0ni 9mo ago

sacs0ni 9mo ago
I built this to have a dedicated wire-protocol client for postgres logical replication. General-purpose SQL clients either don't implement the replication protocol at all, or bury it behind abstractions designed for other use cases. Replication has a bit different mechanics - it's a stateful binary stream requiring LSN tracking, standby heartbeats, and feedback to prevent WAL bloat. Bolting that onto a query-focused client has its own challenges. This is just the transport - raw XLogData frames and LSNs. Use pg_replicate, as an example, if you need "replicate to BigQuery." Use this if you're building replication infrastructure. What it does: - Explicit LSN control - start/stop at specific WAL positions for deterministic recovery - Automatic standby feedback - no more forgotten heartbeats filling your disk with WAL - Bounded channels - backpressure propagates to Postgres naturally - Pure Rust, no libpq What it doesn't do: pgoutput decoding (intentionally). That belongs in a higher layer. Simplest way of using this: while let Some(event) = client.recv().await? { match event { ReplicationEvent::XLogData { wal_end, data, .. } => { process(&data); client.update_applied_lsn(wal_end); } _ => {} } }
- malodyets 9mo ago
  I learned about this tonight when Claude Code picked up your library for my application that uses logical replication. Looking forward to putting it through its paces.
  - sacs0ni 9mo ago
    nice! would appreciate any feedback.
- nkmnz 9mo ago
  > Use pg_replicate, as an example, if you need "replicate to BigQuery." Use this if you're building replication infrastructure. Would I use this if I host my own postgres and want to use replication for „real time backups“ into a hot standby?
  - radimm 9mo ago
    Not OP - but definitely no. In such a case use just physical or - if you have special use case - logical replication that’s built in.
    - sacs0ni 9mo ago
      I agree. pgwire-replication is useful when you need to build a customized and closely controlled pipeline. In fact, it will give you the first part of handling the data (reading from the source), you still need to implement the rest yourself.
- jsjfusua 9mo ago
  [flagged]
gunnarmorling 9mo ago
Nice one, great to see this addition to the Rust ecosystem! Reading through the README, this piqued my curiosity: > Small or fast transactions may share the same WAL position. I don't think that's true; each data change and each commit (whether explicit or not) has its own dedicated LSN. > LSNs should be treated as monotonic but not dense. That's not correct; commit LSNs are monotonically increasing, and within a transaction, event LSNs are monotonically increasing. I.e. the tuple commit-LSN/event-LSN is monotonically increasing, but not LSNs per se. You can run multiple concurrent transactions to observe this.
- sacs0ni 9mo ago
  Good catch, you are correct. I did mix a few things there and the statements were incorrect or at least very misleading. To demo your point I created a gist, for myself and others to see the (commit-LSN, event-LSN) ordering in action: https://gist.github.com/vnvo/a8cf59fc3cd8719dbea56d3bb5201f9b https://gist.github.com/vnvo/a8cf59fc3cd8719dbea56d3bb5201f9... I'll update the readme to reflect this more accurately. Appreciate you taking the time to point it out.

46 points by sacs0ni 9mo ago

I built this to have a dedicated wire-protocol client for postgres logical replication. General-purpose SQL clients either don't implement the replication protocol at all, or bury it behind abstractions designed for other use cases. Replication has a bit different mechanics - it's a stateful binary stream requiring LSN tracking, standby heartbeats, and feedback to prevent WAL bloat. Bolting that onto a query-focused client has its own challenges. This is just the transport - raw XLogData frames and LSNs. Use pg_replicate, as an example, if you need "replicate to BigQuery." Use this if you're building replication infrastructure. What it does: - Explicit LSN control - start/stop at specific WAL positions for deterministic recovery - Automatic standby feedback - no more forgotten heartbeats filling your disk with WAL - Bounded channels - backpressure propagates to Postgres naturally - Pure Rust, no libpq What it doesn't do: pgoutput decoding (intentionally). That belongs in a higher layer. Simplest way of using this: while let Some(event) = client.recv().await? { match event { ReplicationEvent::XLogData { wal_end, data, .. } => { process(&data); client.update_applied_lsn(wal_end); } _ => {} } }

malodyets 9mo ago
I learned about this tonight when Claude Code picked up your library for my application that uses logical replication. Looking forward to putting it through its paces.
- sacs0ni 9mo ago
  nice! would appreciate any feedback.
nkmnz 9mo ago
> Use pg_replicate, as an example, if you need "replicate to BigQuery." Use this if you're building replication infrastructure. Would I use this if I host my own postgres and want to use replication for „real time backups“ into a hot standby?
- radimm 9mo ago
  Not OP - but definitely no. In such a case use just physical or - if you have special use case - logical replication that’s built in.
  - sacs0ni 9mo ago
    I agree. pgwire-replication is useful when you need to build a customized and closely controlled pipeline. In fact, it will give you the first part of handling the data (reading from the source), you still need to implement the rest yourself.
jsjfusua 9mo ago
[flagged]

gunnarmorling 9mo ago

Nice one, great to see this addition to the Rust ecosystem! Reading through the README, this piqued my curiosity: > Small or fast transactions may share the same WAL position. I don't think that's true; each data change and each commit (whether explicit or not) has its own dedicated LSN. > LSNs should be treated as monotonic but not dense. That's not correct; commit LSNs are monotonically increasing, and within a transaction, event LSNs are monotonically increasing. I.e. the tuple commit-LSN/event-LSN is monotonically increasing, but not LSNs per se. You can run multiple concurrent transactions to observe this.

sacs0ni 9mo ago
Good catch, you are correct. I did mix a few things there and the statements were incorrect or at least very misleading. To demo your point I created a gist, for myself and others to see the (commit-LSN, event-LSN) ordering in action: https://gist.github.com/vnvo/a8cf59fc3cd8719dbea56d3bb5201f9b https://gist.github.com/vnvo/a8cf59fc3cd8719dbea56d3bb5201f9... I'll update the readme to reflect this more accurately. Appreciate you taking the time to point it out.