3 ms·
Yes. Kafka is a horizontally-scalable persistent queue (among other things). This sounds kinda pedestrian (like something out of a computer science textbook), b
by wenc 8y ago
Yes. Kafka is a horizontally-scalable persistent queue (among other things). This sounds kinda pedestrian (like something out of a computer science textbook), but people are rapidly discovering uses for it as a systems architecture component.
The concept of "stream-table duality" tells us tables can be thought of as a materialized view of change data operations [INSERT/UPDATE/DELETE] streams. Kafka can be used as a buffer for streaming data that can materialized into a relational table at any time.
One of the more interesting use-cases is multi-target replication: feed your change-data-capture data into Kafka, and replay it on any other backend data store (SQL, Graph Database, NoSQL, etc.)[1]
Conceptually this lets you ingest data into a stream, write the data on multiple backends and keep everything in sync. Martin Klepmann has written a simple (PoC-quality) tool for doing this with Postgres databases.
[1] https://www.confluent.io/blog/bottled-water-real-time-integration-of-postgresql-and-kafka/ https://www.confluent.io/blog/bottled-water-real-time-integr...