4 ms·

Interesting that you bring this up. I looked into heroku's streaming connectors to facilitate an integration with materialize.com, but Heroku's support team wa

Interesting that you bring this up. I looked into heroku's streaming connectors to facilitate an integration with materialize.com, but Heroku's support team wasn't confident we could sync all 187 postgres tables under 1 connection. I thought about using Debezium and Kafka to roll my own micro-batch ETL solution, but listening to this podcast made me walk away slowly: https://www.dataengineeringpodcast.com/datacoral-change-data-capture-episode-177/ https://www.dataengineeringpodcast.com/datacoral-change-data...

gunnarmorling 5y ago
Interesting, what was it from that podcast that made you reconsider? Always eager to learn about opportunities for improving the experience of using Debezium. Disclaimer: I work on Debezium
- tomhallett 5y ago
  Oh wow, by "work on" you mean "the core maintainer of". Thank you for replying. :) The main part I reconsidered based on was the level of effort taking the data from kafka and landing into snowflake, especially around handle postgres schema changes safely. I also have no experience with kafka, so I'd be out of my depth's pretty quickly for a critical part of the architecture. He also expressed the need for building quality checks into the kafka to snowflake code, but those details were a bit sparse (if i recall correctly). Note: all of the above are probably outside the scope of debezium. :) Note 2: your article [1] on using cdc to build audit logs w/ a "transactions" table blew my mind. Once I listened to your data engineering podcast interview [2], I knew there was some implementation of "event sourcing lite w/ a crud app" possible, so I was excited to see you had already laid it out. 1) https://debezium.io/blog/2019/10/01/audit-logs-with-change-data-capture-and-stream-processing/ https://debezium.io/blog/2019/10/01/audit-logs-with-change-d... 2) https://www.dataengineeringpodcast.com/debezium-change-data-capture-episode-114/ https://www.dataengineeringpodcast.com/debezium-change-data-...
  - gunnarmorling 5y ago
    Gotcha, yeah, there's many things to consider indeed when setting up end-to-end pipelines. Thanks for the nice feedback, so happy to hear those resources are useful for folks. As far as event sourcing is concerned, we got another post [1] which might be interesting to you, discussing how "true ES" compares to CDC, pros/cons of either approach, etc. [1] https://debezium.io/blog/2020/02/10/event-sourcing-vs-cdc/ https://debezium.io/blog/2020/02/10/event-sourcing-vs-cdc/

gunnarmorling 5y ago

Interesting, what was it from that podcast that made you reconsider? Always eager to learn about opportunities for improving the experience of using Debezium. Disclaimer: I work on Debezium

tomhallett 5y ago
Oh wow, by "work on" you mean "the core maintainer of". Thank you for replying. :) The main part I reconsidered based on was the level of effort taking the data from kafka and landing into snowflake, especially around handle postgres schema changes safely. I also have no experience with kafka, so I'd be out of my depth's pretty quickly for a critical part of the architecture. He also expressed the need for building quality checks into the kafka to snowflake code, but those details were a bit sparse (if i recall correctly). Note: all of the above are probably outside the scope of debezium. :) Note 2: your article [1] on using cdc to build audit logs w/ a "transactions" table blew my mind. Once I listened to your data engineering podcast interview [2], I knew there was some implementation of "event sourcing lite w/ a crud app" possible, so I was excited to see you had already laid it out. 1) https://debezium.io/blog/2019/10/01/audit-logs-with-change-data-capture-and-stream-processing/ https://debezium.io/blog/2019/10/01/audit-logs-with-change-d... 2) https://www.dataengineeringpodcast.com/debezium-change-data-capture-episode-114/ https://www.dataengineeringpodcast.com/debezium-change-data-...
- gunnarmorling 5y ago
  Gotcha, yeah, there's many things to consider indeed when setting up end-to-end pipelines. Thanks for the nice feedback, so happy to hear those resources are useful for folks. As far as event sourcing is concerned, we got another post [1] which might be interesting to you, discussing how "true ES" compares to CDC, pros/cons of either approach, etc. [1] https://debezium.io/blog/2020/02/10/event-sourcing-vs-cdc/ https://debezium.io/blog/2020/02/10/event-sourcing-vs-cdc/