3 ms·

Event Sourcing advocates: How do you deal with cleaning up events, especially with respect to the GDPR. I inherited an application with 24M+ events and I have

• ComodoHacker 5y ago
  Not an advocate, but you just run the script to clean the data (as ordinary events). It's not really different than handling regular backups. You don't wipe them, just guard them well until they expire.
  • dgb23 5y ago
    Does it comply when I cant see the data but it is still there?
• Fiahil 5y ago
  Two paths : encrypt customer data in payloads with a customer-specific key that you can throw away at will ; OR ; allow some events to rewrite their history / stream.
• oweiler 5y ago
  In Kafka, you can use log-compacted topics, which only keep the last snapshot of a key, and allows deletions via tombstones.
• const314 5y ago
  We don't delete events, just replace sensitive info in events with the specific string marker (by direct update in DB) and then update projections. In this case, the system remains consistent because all the related aggregates and events are still there. You can safely rebuild all the projections without issues and see these string markers instead of deleted info.
  • dsies 5y ago
    ^ 100% this. Do not delete events. They are your source of truth - you shouldn't really even modify them but stripping out PII is "alright". Re 24M+ records: create a batch runner that goes through "jobs" to perform stripping/cleaning tasks. To store state (and to organize cleaners), use a distributed store such as etcd - that way you can bookmark where you were at in the cleaning process.