4 ms·
Keep in mind that a notify performed when nobody is listening is lost. The workers need then to catch up.
by zyngaro 7y ago
Keep in mind that a notify performed when nobody is listening is lost. The workers need then to catch up.
- gunnarmorling 7y agoThis is where the Debezium connector for Postgres [1] comes in: it will retrieve change events from the TX log and push it to brokers such as Apache Kafka or Pulsar, or directly to your application. When not listening, any consumer will continue to read from where it left off before (applying "at least once" semantics). Disclaimer: I work on Debezium [1] https://debezium.io/documentation/reference/0.10/connectors/postgresql.html https://debezium.io/documentation/reference/0.10/connectors/...
- makkesk8 7y agoI fail to see how Debezium would solve this as it's just as likely Debezium wouldn't be watching the WAL as if you had a minimal number of workers always listening and those went down for some reason. Could you please elaborate?
- gunnarmorling 7y agoDebezium stores the WAL position it has read ("offset"). So while it's down, of course it won't be able to read any events. But once it's restarted, it will continue to read the WAL from the last offset. So no events will be missed also when the connector isn't running for some time (WAL segments are retained by the DB until acknowledged by a logical decoding client such as Debezium).