4 ms·

This is very obvious. Synchronization based on sequential change log is brittle by nature.

UltraSane 1y ago
What are the most common failure modes? Whenever I implement CDC I always try to implement some kind of integrity check that runs at some reasonable interval that can detect and try to fix any discrepancies. CDC is by far the most efficient form of data synchronization in most situations.
- sakesun 1y ago
  You are doing it right for having periodic check. Why you would want to know what common failure modes are then ?
  - phanimahesh 1y ago
    Intellectual curiosity and a desire to learn something, to exchange ideas and learn if there's something they haven't thought of, etc?
  - UltraSane 1y ago
    not having a error detection method is like using a stepper motor with open-loop control. Not something any good engineer will accept.
- halfcat 1y ago
  The point is, you can’t do synchronization reliably using any message/event-based approach alone. You always need a reconciliation mechanism, as you’ve correctly noticed. It’s always shocking to me how many FAANG people will say, “we want an event driven solution with a message bus, that the right way to do it, we don’t want batch, that can’t scale”, and then need to bolt on a validation/reconciliation step for it to be reliable. Which of course is a batch job. Unless you control a system end to end (which is rare, there’s usually some data from a system you don’t control the schema of), or are highly incentivized to make sync happen reliably (e.g. bitcoin), you’re always limited by a batch job somewhere in the system. You could even say the batch job is often doing the heavy lifting, and the message bus is just an optimization (that’s often not adding the value needed to justify the complexity).
  - UltraSane 1y ago
    Using something like Kafka you can get reliable at least once messaging and then you just need to make the CDC updates idempotent. I'm not sure exactly what you mean by batch job but if n bytes change at the source you shouldn't have to copy more than n bytes to the destination.
    - halfcat 1y ago
      If you can get all of your data efficiently (and transactionally consistent) into Kafka, that’s the scenario I mention where you have control of all your systems. Inevitably, even if you achieve this at some point, it never lasts. Your company acquires another, or someone pushes for a different HR/CRM/whatever system and gets it. You mention if n bytes change in the source, but many systems have no mechanism of determining that n bytes have changed without scanning the entire data set. So we’re back to a batch job (cron, or similar).
      - UltraSane 1y ago
        "many systems have no mechanism of determining that n bytes have changed without scanning the entire data set." This is so insanely inefficient it can't scale to very large amounts of data. If you can't do data syncing at the application layer you can do it at the storage layer with high end storage arrays that duplicate all writes to a second storage array, either synchronously or asynchronously. Or duplicate snapshots to a remote array. They work really well.
- datadrivenangel 1y ago
  Integrity is the hard part. Usually hard deletes in the source system are hard to propagate well because full deletion is awkward when you only expect to *update* data.

What are the most common failure modes? Whenever I implement CDC I always try to implement some kind of integrity check that runs at some reasonable interval that can detect and try to fix any discrepancies. CDC is by far the most efficient form of data synchronization in most situations.

sakesun 1y ago
You are doing it right for having periodic check. Why you would want to know what common failure modes are then ?
- phanimahesh 1y ago
  Intellectual curiosity and a desire to learn something, to exchange ideas and learn if there's something they haven't thought of, etc?
- UltraSane 1y ago
  not having a error detection method is like using a stepper motor with open-loop control. Not something any good engineer will accept.
halfcat 1y ago
The point is, you can’t do synchronization reliably using any message/event-based approach alone. You always need a reconciliation mechanism, as you’ve correctly noticed. It’s always shocking to me how many FAANG people will say, “we want an event driven solution with a message bus, that the right way to do it, we don’t want batch, that can’t scale”, and then need to bolt on a validation/reconciliation step for it to be reliable. Which of course is a batch job. Unless you control a system end to end (which is rare, there’s usually some data from a system you don’t control the schema of), or are highly incentivized to make sync happen reliably (e.g. bitcoin), you’re always limited by a batch job somewhere in the system. You could even say the batch job is often doing the heavy lifting, and the message bus is just an optimization (that’s often not adding the value needed to justify the complexity).
- UltraSane 1y ago
  Using something like Kafka you can get reliable at least once messaging and then you just need to make the CDC updates idempotent. I'm not sure exactly what you mean by batch job but if n bytes change at the source you shouldn't have to copy more than n bytes to the destination.
  - halfcat 1y ago
    If you can get all of your data efficiently (and transactionally consistent) into Kafka, that’s the scenario I mention where you have control of all your systems. Inevitably, even if you achieve this at some point, it never lasts. Your company acquires another, or someone pushes for a different HR/CRM/whatever system and gets it. You mention if n bytes change in the source, but many systems have no mechanism of determining that n bytes have changed without scanning the entire data set. So we’re back to a batch job (cron, or similar).
    - UltraSane 1y ago
      "many systems have no mechanism of determining that n bytes have changed without scanning the entire data set." This is so insanely inefficient it can't scale to very large amounts of data. If you can't do data syncing at the application layer you can do it at the storage layer with high end storage arrays that duplicate all writes to a second storage array, either synchronously or asynchronously. Or duplicate snapshots to a remote array. They work really well.
datadrivenangel 1y ago
Integrity is the hard part. Usually hard deletes in the source system are hard to propagate well because full deletion is awkward when you only expect to *update* data.