4 ms·
> rolling back the DB transaction if publishing fails. If your DB transaction fails after the Kafka message has been sent, you'll end up with inconsistencies b
by gunnarmorling 6y ago
> rolling back the DB transaction if publishing fails.
If your DB transaction fails after the Kafka message has been sent, you'll end up with inconsistencies between your local database state and what has been published externally via Kafka. That's exactly the kind of dual write issues the outbox pattern suggested in my other comment avoids.
> unless you are sending this data to the DB
You actually can do this, and still keep this metadata separately from your domain tables. The idea is to add this metadata into a separate table, keyed by transaction id, and then use downstream stream processing to enrich the actual change events with that metadata. I've blogged about an approach for this a while ago here: https://debezium.io/blog/2019/10/01/audit-logs-with-change-data-capture-and-stream-processing/ https://debezium.io/blog/2019/10/01/audit-logs-with-change-d....
- rclayton 6y agoWe don’t use auto commits with Kafka so it’s not a problem for us. Of course, this does reduce our publishing throughput - though that’s not a problem for us at our current scale. Either way, I’m intrigued by the outbox pattern. As long as it is transparent to developers, it does seem like an ideal CDC mechanism.