4 ms·

Well written post! However I'd like note that, we commonly hear from users that Debezium for Postgres CDC is hard to setup, manage and less flexible around data

Well written post! However I'd like note that, we commonly hear from users that Debezium for Postgres CDC is hard to setup, manage and less flexible around data format (only json and avro). It requires at least a few months of effort to put to production. We at PeerDB (https://github.com/PeerDB-io/peerdb https://github.com/PeerDB-io/peerdb) are trying to solve this problem. We already have a connector out for Azure Event Hubs https://blog.peerdb.io/enterprise-grade-replication-from-postgres-to-azure-event-hubs https://blog.peerdb.io/enterprise-grade-replication-from-pos... This blog captures how we are building a more production/enterprise ready solution to Postgres CDC. Connectors for Kafka, Redpanda, PubSub are already merged to main and we plan to GA them soon!

gunnarmorling 3y ago
Debezium supports all kinds of data formats. For instance, ProtoBuf is often used these days, but it's easily customizable/configurable to have other formats, too. > "At least a few months" Can you back this up? This doesn't match my experience from working with users in the Debezium community at all. Don't get me wrong, Debezium certainly has a learning curve, but that's nearwhere realistic. I think you're doing interesting stuff at PeerDB, but it would be nice if you could do without this kind of unfounded anti-Debezium FUD.
- saisrirampur 3y ago
  Thanks for the feedback here! My apologies if this came across as FUD. So let me clarify, Debezium is proven. Many large companies use Debezium for production/enterprise grade CDC. So it is indeed a great piece of software!! However, higher Capex and Opex costs to put Debezium to prod is one of the common problem we've heard from Postgres users. This is indeed related to the learning curve: One issue we've heard is the emphasis on command-line interface (vs UI) which provides a bunch of options for configurability but makes it complex for a first time (average) user to work with. There is Debezium UI, but that is not the default recommendation (and seems to be still in incubation). At PeerDB, we are trying to address this by working on a simple (yet Advanced) UI and Postgres-compatible SQL layer (for more complex pipelines) which we believe is more intuitive than bash scripts. Disclaimer: PeerDB is still in its early stages, and we have yet to support the full range of configurability options that Debezium offers. We may encounter same challenges as Debezium related to learning curve. However, we plan keep "Usability" as top priority while building software. So far, with a few production customers, the direction we are taking seems positive. In regards to formats, we have a few users who wanted Flatbuffers/MsgPack (binary JSON) formats and it wasn't trivial to setup with Debezium. These are users who haven't worked with Debezium before but after few days of effort, felt that it wasn't very easy. Thanks again for the feedback here! And apologies if my comment came across negative. Thanks for probing me to clarify what I meant. :)
deleted 3y ago
[deleted]

gunnarmorling 3y ago

Debezium supports all kinds of data formats. For instance, ProtoBuf is often used these days, but it's easily customizable/configurable to have other formats, too. > "At least a few months" Can you back this up? This doesn't match my experience from working with users in the Debezium community at all. Don't get me wrong, Debezium certainly has a learning curve, but that's nearwhere realistic. I think you're doing interesting stuff at PeerDB, but it would be nice if you could do without this kind of unfounded anti-Debezium FUD.

saisrirampur 3y ago
Thanks for the feedback here! My apologies if this came across as FUD. So let me clarify, Debezium is proven. Many large companies use Debezium for production/enterprise grade CDC. So it is indeed a great piece of software!! However, higher Capex and Opex costs to put Debezium to prod is one of the common problem we've heard from Postgres users. This is indeed related to the learning curve: One issue we've heard is the emphasis on command-line interface (vs UI) which provides a bunch of options for configurability but makes it complex for a first time (average) user to work with. There is Debezium UI, but that is not the default recommendation (and seems to be still in incubation). At PeerDB, we are trying to address this by working on a simple (yet Advanced) UI and Postgres-compatible SQL layer (for more complex pipelines) which we believe is more intuitive than bash scripts. Disclaimer: PeerDB is still in its early stages, and we have yet to support the full range of configurability options that Debezium offers. We may encounter same challenges as Debezium related to learning curve. However, we plan keep "Usability" as top priority while building software. So far, with a few production customers, the direction we are taking seems positive. In regards to formats, we have a few users who wanted Flatbuffers/MsgPack (binary JSON) formats and it wasn't trivial to setup with Debezium. These are users who haven't worked with Debezium before but after few days of effort, felt that it wasn't very easy. Thanks again for the feedback here! And apologies if my comment came across negative. Thanks for probing me to clarify what I meant. :)

deleted 3y ago

[deleted]