5 ms·

Is Debezium the only good CDC tool out there? I have a fairly simple data stack and am looking at integrating a CDC solution but I really dont want to touch Ka

• moltar 2y ago
  AWS has DMS with a serverless option which is great for CDC
  • nxm 2y ago
    Except when a new column is added upstream and DMS inserts the new column not at the end but before the dms timestamp fields, ruining the index based loads of DMS files
• nchmy 2y ago
  Conduit.io is where it's at. FAR more source and destination connectors, easier to deploy etc... Pair it with NATS to replace Kafka's mess
• rockwotj 2y ago
  Redpanda Connect is a yaml file that has a cdc input and you can process the data and send it anywhere, no Kafka required. I’ve seen postgres cdc directly being written to Snowflake. https://docs.redpanda.com/redpanda-connect/components/inputs/postgres_cdc/ https://docs.redpanda.com/redpanda-connect/components/inputs...
• gunnarmorling 2y ago
  You don't need Kafka in order to run Debezium. While that certainly is the most common deployment, you can use Debezium also with things like Kinesis, Google Cloud Pub/Sub, or even plain HTTP sinks, all via Debezium Server (https://debezium.io/documentation/reference/stable/operations/debezium-server.html https://debezium.io/documentation/reference/stable/operation...). You even can embed it as a library into your application for the highest degree of flexibility. That said, streaming events into Kafka (or something like it), rather than pushing change events directly to destinations like Snowflake has big advantages when it comes to reprocessing data, setting up additional consumers, etc.