2 ms·
Dealt with something like this when I joined a previous job years ago, Kafka was the main data store, microservices had their state in memory they would build a
by dgellow 3y ago
Dealt with something like this when I joined a previous job years ago, Kafka was the main data store, microservices had their state in memory they would build at startup by processing their whole history of events (that was ridiculously slow for some of them). Then GDPR came in place and the whole “keep everything in Kafka forever” had to face the reality of “not allowed to keep PII for longer than 30 days” :)
(Before someone suggests it, no messages weren’t encrypted, just throwing the key away wasn’t an option)
- jabradoodle 3y agoWould be expensive but you can do a copy and replace to keep the data you need, on a new topic. If you don't need the data, then you don't need it.
- andsbf 3y ago> ... they would build at startup by processing their whole history of events (that was ridiculously slow for some of them) There is a basic technique to solve this, you snapshot every "nth" event