3 ms·
Don't you always need a database after reading events from Kafka to deduplication? So the competing solutions are: PostgreSQL or Kafka+PostgreSQL Kafka does p
by gmokki 11mo ago
Don't you always need a database after reading events from Kafka to deduplication?
So the competing solutions are:
PostgreSQL or Kafka+PostgreSQL
Kafka does provide some extrs there, handling load spikes, more clients that PG can handle natively and resilience to some DB downtime. But is it worth the complexity, in most cases no.
- jevyjevjevs 11mo agoActually you can avoid having a separate DB! You can build a materialized view of the data using [KTables](https://developer.confluent.io/courses/kafka-streams/ktable/ https://developer.confluent.io/courses/kafka-streams/ktable/) or use [interactive queries](https://developer.confluent.io/courses/kafka-streams/interactive-queries/ https://developer.confluent.io/courses/kafka-streams/interac...). The "table" is built up from a backing kafka topic so you don't need maintain another datastore if the data view you want is entirely derived from one or more Kafka topics.
- enether 11mo agoMy understanding is this is pretty niche and can be complex.