4 ms·

Four years ago I heard "Kafka IS your database". I thought maybe these insane people (probably parroting some tech company enterprise penetration propaganda e.

Four years ago I heard "Kafka IS your database". I thought maybe these insane people (probably parroting some tech company enterprise penetration propaganda e.g. Confluent) would have a better story, but... no. Anyway, yeah, sure, keep logs. But a lot of that article about commands and events is something that only exists if you had one ubiquitous language, system, and OS. You know, the almost literal "seamless" where there aren't any seams. Sure that will probably plug into some enterprise bus and enterprise integration and enterprise ... anyway. Competent developers will understand what events to preserve and log and possibly allow retry/repeats. Anyone who has looked at a Splunk bill will realize that just storing all the logs everywhere is very expensive, which is another way of saying "wasteful". But any generic enterprisey event system will basically start and end with splunk-level log aggregation and kinda-analysis.

dgellow 3y ago
Dealt with something like this when I joined a previous job years ago, Kafka was the main data store, microservices had their state in memory they would build at startup by processing their whole history of events (that was ridiculously slow for some of them). Then GDPR came in place and the whole “keep everything in Kafka forever” had to face the reality of “not allowed to keep PII for longer than 30 days” :) (Before someone suggests it, no messages weren’t encrypted, just throwing the key away wasn’t an option)
- jabradoodle 3y ago
  Would be expensive but you can do a copy and replace to keep the data you need, on a new topic. If you don't need the data, then you don't need it.
- andsbf 3y ago
  > ... they would build at startup by processing their whole history of events (that was ridiculously slow for some of them) There is a basic technique to solve this, you snapshot every "nth" event

dgellow 3y ago

Dealt with something like this when I joined a previous job years ago, Kafka was the main data store, microservices had their state in memory they would build at startup by processing their whole history of events (that was ridiculously slow for some of them). Then GDPR came in place and the whole “keep everything in Kafka forever” had to face the reality of “not allowed to keep PII for longer than 30 days” :) (Before someone suggests it, no messages weren’t encrypted, just throwing the key away wasn’t an option)

jabradoodle 3y ago
Would be expensive but you can do a copy and replace to keep the data you need, on a new topic. If you don't need the data, then you don't need it.
andsbf 3y ago
> ... they would build at startup by processing their whole history of events (that was ridiculously slow for some of them) There is a basic technique to solve this, you snapshot every "nth" event