4 ms·
Sounds like a great use case for log-based change data capture actually. That way, you won't miss any updates also in case of outages, network issues, etc. Once
by gunnarmorling 5y ago
Sounds like a great use case for log-based change data capture actually. That way, you won't miss any updates also in case of outages, network issues, etc. Once everything is up and running again, such pipeline would resume from where it left off before. Plus, (unlike anything based on application-side triggers) no need to modify your app or the risk of missing changes e.g. done directly on the DB via SQL.
Disclaimer: I work on Debezium, an open-source CDC solution
- supermatt 5y agoSo debezium is offering a nice way to listen to the replication logs? Awesome. Will have to check that out :)
- gunnarmorling 5y agoYes, exactly. Debezium (debezium.io) provides log-based CDC for a variety of databases, exposing a unified event format as much as possible. It can be used with Kafka (Connect), but also stand-alone or as a library embedded into your app.