7 ms·
Great article. One thing that wasn't pointed out that might be of interest to someone learning about Event Sourcing is that it introduces some challenges if you
by xtagon 7y ago
Great article. One thing that wasn't pointed out that might be of interest to someone learning about Event Sourcing is that it introduces some challenges if you are to be compliant with GDPR and similar laws. For example, if your event log is immutable, and you use it as an audit log, then by nature you are not ever deleting data. There are solutions to this (for example, crypto-erasure), but it can be non-trivial to implement.
- tunesmith 7y agoI saw the Akka people talking about this on twitter once, I think they were theorizing that encrypting the data in the log would be sufficient, because then "deleting the key" could be interpreted as the deletion of the record (even though the useless data still exists in the log). But I'm not sure this was ever legally validated?
- guywhocodes 7y agoThat is a very interesting approach, however I can imagine that it can be quite a large database of encryption keys. I'll have to build a small system for this and try it.
- imglorp 7y agoOh, why? Holding one key for every human on earth would fit in 8 TB plus read-mostly replicas. You'd delete their key if they made GDPR removal request and the keyless, encrypted, immutable entries would be entombed in place.
- pookeh 7y agoWhat do you do when the data involves multiple people?
- shrimpx 7y agoIt's unclear (regardless of crypto shredding) what the delete policy should be if the data involves multiple people.
- randolorian 7y agoWhy would it? Typically you would have one stream per user.
- tunesmith 7y agoEvent: "User A sent Message X to User B". (I think the answer here is again to have PII be in other repositories, and only refer to them by id in the log.)
- specialist 7y agoGood question. Create alias proxies?
- xtagon 7y agoThat is what I was referring to as "crypto-erasure" (also known as "crypto-shredding"). I'm not sure what counts as legally validated, but some have shared concerns that the encryption you use to do this would need to be future-proof against, say, advancements in quantum computing cracking the encryption down the road even after the key has been thrown out.
- rodocite 7y agoif the references don't point to actual data, you don't need this. - minimum 2 parts. a relay (reference hash) and the cold/true storage portion. you can break the reference hash up and reassemble only for secret holders. which brings us to: - content-based routing - there are also deterministic vaults for rolling keys i'm actually not sure in what high-level situation "crypto-erasure" would work in because being able to re-key a reference means you have complete control. so why would you need to erase the "bad" key when you can just switch the reference? Eth draft for enabling cold storage relay https://github.com/ethereum/EIPs/blob/master/EIPS/eip-1077.md https://github.com/ethereum/EIPs/blob/master/EIPS/eip-1077.m... Decentralized ID https://www.w3.org/TR/did-core/ https://www.w3.org/TR/did-core/ ^ both generally use the same concept i described above and solve GDPR "delete" issue. actually, it solves GDPR completely if you can just rely on the DID. "hard delete" is a separate issue, though-- no other way to get around that but to fork/version + replay your store and re-reference anyone who wants to hard delete if you didn't use reference hashes.
- specialist 7y agoJust like password stores. Store salted and hashed password value, vs actual password. Cleverly applying this strategy to all sensitive datas is the Translucent Database thesis.
- rodocite 7y agoI'm pretty sure you can "soft-delete" for GDPR compliance. So this concern is sort of a non-issue. Besides that: 1. If you're using an immutable structure, as long as you use references, you can obfuscate data. Blockchains ran into this problem before GDPR requirements and that's essentially all they do. 2. ^ "Update" strategy for event sourcing is the same as above. Essentially a copy of the log or the log slice then a re-indexing to remove/update streams, events, projections, etc. Greg Young talks about the indexing internals (for Event Store) in the 2012 video.
- nchase 7y agoCan you describe more precisely how you're defining "soft-delete" here?
- rodocite 7y agodata is still stored somewhere but any routing to the data is disabled and the entity is disassociated.
- GordonS 7y agoCould you point towards an authoritative source for this claim? As a consumer, if I request deletion of my data, I expect it to be actually deleted - not just have a "deleted" flag set. With soft-deletion, the data is still right there, ready to be abused after a breach.
- lawik 7y agoSoft delete in the sense of removing availability but keeping the data shouldn't be GDPR compliant from what I've seen. Not in regards to right to be forgotten and restrictions on keeping PII.
- the_gipsy 7y ago"soft-delete" is not GDPR compliant.
- codebje 7y agoYour event log should IMO be nominally immutable rather than actually immutable. You should feel free to take actions such as expunging private or sensitive data as appropriate. Keep the events, but rewrite them to contain only the desired data. Trivial to implement, and simple. I'd only worry about stuff like crypto-erasure if you physically cannot alter the past, such as if you have a requirement for non-repudiation or some such. Doing it just for technical purity isn't worth the cost :-)
- andreareina 7y agoI think cases where you cannot alter the past (or can only do so with difficulty) are fairly common; backups will easily tend to fall in that category.
- TheCoelacanth 7y agoBackups have the same GDPR concerns regardless of whether you are doing event sourcing or not.
- eweise 7y agoDon't stick PII in the events. Write it to a dedicated PII db and reference it in the event. Easy to get rid of the person's data. Just delete from the PII db.