4 ms·

Flink CDC Connectors

28 points by maydemir 5y ago

glogla 5y ago
I like the idea of using CDC for data pulls, but there are some issues with it. 1/ On the target, you have to do compaction, deduplication, whatever you call the reconstruction of "latest state" as that's usually what people need, which is typically pretty computationally expensive. 2/ I worry about the operational aspects of it. How often does CDC break and needs resync? What happens if someone runs alter table? And when it breaks, do I need cooperation of the admin team to get it running again? Stupid periodic "select *" (with something like Stitch or Airbyte or whatever) needs very little cooperation from the source owner and are pretty robust. Does anyone have any experience here?
- WJW 5y ago
  Periodic SELECT queries seems like it would create big issues with transaction length for big enough tables (say ~1 billion rows and larger), because the transaction stays open as long as the select query is not complete. This will definitely cause issues on busy tables. The operational aspects are pretty mild IMO, it's just another read replica from the view of the database. You indeed have to be careful with schema changes and admin, but that is something you already have to handle if you are running Flink. So I'd say that for big databases, CDC is not all that bad. And if your database is still small enough that you can get away with "SELECT *" a few times per day, why are you using Flink in the first place?
- xyzzy_plugh 5y ago
  There's no silver bullet. Ultimately you're responsible for copying records from a source, no tool can magically solve that for you. If you change your schema or need to backfill, it's good to have strong tools and frameworks to handle that. Ultimately data pipelines are code, and need to be treated as code. If a library or API changes, you make the associated changes required. The problem with all of these tools, CDC or not, select * or not, is they still just punt the problem to you. Instead, I've had wild success building tooling from first principles to fetch records in batches or streams, as a first class API rather than a backdoor hack.
- gunnarmorling 5y ago
  There's several advantages of log-based CDC (as provided by Debezium, upon which the Flink CDC connectors are based), such as much lower latency, never missing any changes, ability to capture deletes, no impact on data models, etc. I've blogged about the differences between log-based and query-based CDC a while ago [1]. One advantage of query-based is that its entry barrier is lower, if there's no log-based connector for a given database, you'll have to go for query-based. Disclaimer: I work on Debezium [1] https://debezium.io/blog/2018/07/19/advantages-of-log-based-change-data-capture/ https://debezium.io/blog/2018/07/19/advantages-of-log-based-...

28 points by maydemir 5y ago

I like the idea of using CDC for data pulls, but there are some issues with it. 1/ On the target, you have to do compaction, deduplication, whatever you call the reconstruction of "latest state" as that's usually what people need, which is typically pretty computationally expensive. 2/ I worry about the operational aspects of it. How often does CDC break and needs resync? What happens if someone runs alter table? And when it breaks, do I need cooperation of the admin team to get it running again? Stupid periodic "select *" (with something like Stitch or Airbyte or whatever) needs very little cooperation from the source owner and are pretty robust. Does anyone have any experience here?

WJW 5y ago
Periodic SELECT queries seems like it would create big issues with transaction length for big enough tables (say ~1 billion rows and larger), because the transaction stays open as long as the select query is not complete. This will definitely cause issues on busy tables. The operational aspects are pretty mild IMO, it's just another read replica from the view of the database. You indeed have to be careful with schema changes and admin, but that is something you already have to handle if you are running Flink. So I'd say that for big databases, CDC is not all that bad. And if your database is still small enough that you can get away with "SELECT *" a few times per day, why are you using Flink in the first place?
xyzzy_plugh 5y ago
There's no silver bullet. Ultimately you're responsible for copying records from a source, no tool can magically solve that for you. If you change your schema or need to backfill, it's good to have strong tools and frameworks to handle that. Ultimately data pipelines are code, and need to be treated as code. If a library or API changes, you make the associated changes required. The problem with all of these tools, CDC or not, select * or not, is they still just punt the problem to you. Instead, I've had wild success building tooling from first principles to fetch records in batches or streams, as a first class API rather than a backdoor hack.
gunnarmorling 5y ago
There's several advantages of log-based CDC (as provided by Debezium, upon which the Flink CDC connectors are based), such as much lower latency, never missing any changes, ability to capture deletes, no impact on data models, etc. I've blogged about the differences between log-based and query-based CDC a while ago [1]. One advantage of query-based is that its entry barrier is lower, if there's no log-based connector for a given database, you'll have to go for query-based. Disclaimer: I work on Debezium [1] https://debezium.io/blog/2018/07/19/advantages-of-log-based-change-data-capture/ https://debezium.io/blog/2018/07/19/advantages-of-log-based-...