3 ms·
You could keep using Postgres, but in a special way for such log tables: insert and delete the log records right away. Then use a change data capture tool like
by gunnarmorling 6y ago
You could keep using Postgres, but in a special way for such log tables: insert and delete the log records right away. Then use a change data capture tool like Debezium to capture changes from the transaction log and stream them to Kafka, or any other destination.
Your log table will always be empty (so not taking up disk space), but the CDC tool will be able to get all the information from the TX logs (which themselves can be discarded after all replication slots have advanced a given offset). The advantage is that your log entries are transactionally written with the actual application data itself.
We use this technique for instance in the Debezium Quarkus extension for implementing the outbox pattern [1]. Postgres even allows you to insert records directly to the TX log via pg_logical_emit_message() - so no table involved at all -, but we don't support this in Debezium yet.
Disclaimer: I work on Debezium.
[1] https://debezium.io/documentation/reference/integrations/outbox.html https://debezium.io/documentation/reference/integrations/out...
- jaytaylor 6y agoThis looks interesting, looking forward to reading up on it. Would be really cool if it worked with SQLite ;) Any chance JDK11 support will come for the Debezium Java core [1]? [1] https://github.com/debezium/debezium https://github.com/debezium/debezium
- gunnarmorling 6y agoFunny you should ask, we were just talking about raising the minimum baseline to JDK 11 the other day (currently it still is compatible with Java 1.8). That said, you definitely can use Debezium with Java 11 already; so perhaps you an entail a bit what exactly you mean by JDK 11 support here? Re SQLite, is there any CDC interface which could be used to implement a connector?
- jaytaylor 6y agoGot you, that sounds great! When the README listed Java 1.8 as a requirement, I assumed (perhaps incorrectly) an implication that it works with _only_ Java 1.8. At my work, some critical libraries only work with Java 8 or 11, so the transition from 8 was challenging on the dependency versioning front. It's a gigantic project and codebase, though. Re: SQLite, not that I know of, but I don't have deep knowledge on the matter. HN may go crazy (in a good way!) if there were a way to pull this off though!!!