3 ms·

With all these CDC solutions based on logical replication, how are PostgreSQL failovers handled? You can't really rewind ClickHouse. Does it leave it with incon

• saisrirampur 2y ago
  Planned failovers can be managed through maintenance windows by recreating replication slots before maintenance ends. Unplanned failovers require resyncs, which are heavily optimized with PeerDB (https://blog.peerdb.io/parallelized-initial-load-for-cdc-based-streaming-from-postgres#heading-parallelized-initial-snapshot-for-cdc-based-streaming https://blog.peerdb.io/parallelized-initial-load-for-cdc-bas...). With PG17+, this shouldn’t be an issue due to failover slots. Another important observation: Aurora Postgres, which is used by many customers, persists slots during failover, so this isn’t a problem at all. PeerDB has built-in retries that resume from the last committed source LSN.
• higeorge13 2y ago
  I guess you talk about hard failures or replication slot deletion, right? Debezium has a snapshot mode (1) to refetch everything from scratch, we have tested it but opted out of using it because it would put pressure on postgres with the queries on huge tables and on kafka and its storage. Imo we prefer to manually rebuild the clickhouse tables via postgres snapshots and latest cdc data, but there might be better options or ideas. (1) https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-snapshots https://debezium.io/documentation/reference/stable/connector...