3 ms·
I think the best pattern for this is Change Data Capture (CDC). Checkout out something like Debezium. In short, IIRC, it looks into the RDBMSs redo log, and tu
by bradwood 5y ago
I think the best pattern for this is Change Data Capture (CDC). Checkout out something like Debezium.
In short, IIRC, it looks into the RDBMSs redo log, and turns it into a message stream. You can then stream this into some kind of messaging platform (Kakfa is typical) and then process the data any way you want. Log it, aggregate it, archive it, stick it in another database, or object store, etc...
- emptysea 5y agoHow would CDC include information like the user making the change?
- shaeqahmed 5y agoA CDC alone wouldn't. You would need to track and store that metadata in an additional database along with transaction ID, so it can be joined with the raw change stream to form a complete audit log.
- gunnarmorling 5y agoWrote about this very approach on the Debezium blog some time ago: https://debezium.io/blog/2019/10/01/audit-logs-with-change-data-capture-and-stream-processing/ https://debezium.io/blog/2019/10/01/audit-logs-with-change-d.... It uses Kafka Streams for enriching actual change events with that transaction-scoped metadata.