4 ms·

What is Change Data Capture (CDC) ? Peerdb doesn't seem to inform on the core of the problem it solves, here's a reference from Debezium (mentioned in the text

What is Change Data Capture (CDC) ? Peerdb doesn't seem to inform on the core of the problem it solves, here's a reference from Debezium (mentioned in the text) > set up and configure Debezium to monitor your databases, and then your applications consume events for each row-level change made to the database. Only committed changes are visible, so your application doesn't have to worry about transactions or changes that are rolled back. It's good to know! This model seems to turn row changes into effectively a side-effect invocation for a queue.

treyfitty 2y ago
To my knowledge, CDC provides a solution to the following 2 pain points. Would love to hear more: 1. Replication: Imagine you have 5 different analytical environments. Do you backup & restore to each destination, essentially re-writing 10TB of data, when only 2MB worth of data changed? You'd be surprised how many organizations still do this. 2. System Triggering: How can you inform a downstream system an event occurred? There are many mechanisms to do so, and they vary in complexity, but given the requirements are simple (if a row in the DB changes, let's tag this change as an event and just pass it to all the downstream systems that need to know this)
- assaxor 2y ago
  Additionally, it can help with systems where 2-phase commits can lead to weird and complex situations. This is where applying SAGA and using Transactional Outbox can help: it should allow for abstracting away the event stream away from the main application logic, the CDC mirror can then handle generating the stream. This can also allow for plug-and-play of other downstream services/sinks from the CDC stream.
saisrirampur 2y ago
Change Data Capture (and its use cases) is a well known topic in the data-engineering and streaming communities. Hence we didn't mention more about it. But we will add it to our future posts. Thanks for the feedback here! Here goes some commentary on CDC use-cases: CDC provides a reliable stream of database changes, enabling various use cases such as real-time alerting (e.g., fraud detection in banking), replicating OLTP data to other types of stores (OLAP, search, etc.) for real-time analytics and search, and implementing the transactional outbox pattern which exposes database changes to other microservices/apps (e.g., a bank exposing a raw feed of transactions to its rewards microservice, etc.).

treyfitty 2y ago

To my knowledge, CDC provides a solution to the following 2 pain points. Would love to hear more: 1. Replication: Imagine you have 5 different analytical environments. Do you backup & restore to each destination, essentially re-writing 10TB of data, when only 2MB worth of data changed? You'd be surprised how many organizations still do this. 2. System Triggering: How can you inform a downstream system an event occurred? There are many mechanisms to do so, and they vary in complexity, but given the requirements are simple (if a row in the DB changes, let's tag this change as an event and just pass it to all the downstream systems that need to know this)

assaxor 2y ago
Additionally, it can help with systems where 2-phase commits can lead to weird and complex situations. This is where applying SAGA and using Transactional Outbox can help: it should allow for abstracting away the event stream away from the main application logic, the CDC mirror can then handle generating the stream. This can also allow for plug-and-play of other downstream services/sinks from the CDC stream.

saisrirampur 2y ago

Change Data Capture (and its use cases) is a well known topic in the data-engineering and streaming communities. Hence we didn't mention more about it. But we will add it to our future posts. Thanks for the feedback here! Here goes some commentary on CDC use-cases: CDC provides a reliable stream of database changes, enabling various use cases such as real-time alerting (e.g., fraud detection in banking), replicating OLTP data to other types of stores (OLAP, search, etc.) for real-time analytics and search, and implementing the transactional outbox pattern which exposes database changes to other microservices/apps (e.g., a bank exposing a raw feed of transactions to its rewards microservice, etc.).