6 ms·

Okay, this approach is really cool, but... just hear me out: what if... you don't store personally identifiable data in the event store? What if you only store

by arpa 9y ago

Okay, this approach is really cool, but... just hear me out: what if... you don't store personally identifiable data in the event store? What if you only store references/ids that point to services which can resolve those ids to data, and that data needs not to be immutable. In the event of erasure request you shred/anonymize that data only, without touching kafka. I mean it's pretty obvious to me, to the point i feel like i'm missing some huge point and grossly misunderstanding the whole problem

jshen 9y ago
The whole point of event sourcing is the ability to manage state effectively. When you move state out of the event sourcing model you are losing the state management benefits of event sourcing for that data. Maybe that’s a reasonable trade-off for GDPR, but there is a trae-off.
- ithkuil 9y ago
  There are ways to put some state outside the event stream that play nicely with the model. For example you could store the body in a content addressed store (a key value store whose keys are hashes of the value). This preserves the ideal model of immutable events as you cannot accidentally change the content of the event since that would violate the property of the CAS. But nothing prevents you from deleting an entry in the CAS. However you still have to find all the records you need to delete! The trick with forgetting the encryption key let makes this step a O(1) operation. On the other hand it forces you to decide upfront the granularity of ownership, i.e. if you treat payload deletion as a batch process you can deal with mistakes, change your mind about which data should or should not be deleted. While with the encryption technique, changing your mind would require you to perform a full history replay and reencrypt data in a new stream (and throw away the old one) with the new granularity model.
michaelbuckbee 9y ago
This is actually a recommended approach within the GDPR legislation itself Article 25: "Pseudonymization" https://blog.varonis.com/gdpr-requirements-list-in-plain-english/#article25 https://blog.varonis.com/gdpr-requirements-list-in-plain-eng...
dnomad 9y ago
I think that's exactly what any real system would do. This sort of widespread event encryption and forgetting is crazy. It is a very bad idea to delete events from the immutable event log. There is no telling how downstream consumers will react but it probably won't be good. It's much simpler to only hold references to a user (UserID) and then if you need to forget that a user you simply burn whatever is behind that UserID. This is actually how the GDPR is designed: it's not about user data it's about personal data, that is data that can be associated or related back to an "identifiable person." The problem is not that you need to delete all the user's data across all your systems, the problem is that you need to break any associations that would allow you to identify a natural person who has asked to be "forgotten." The funny thing is that while some people make lots of noises about GDPR being some huge "burden," the reality is that any architect worth her salt should've been designing systems like this from the very start rather than letting personal data be replicated en mass from one system to another. It is a basic normalization of data. All the GDPR is doing, like most regulations, is requiring businesses follow best practices and not cut corners that might harm end users. The great thing about this in the long run is that this fixes a huge problem with the internet. Today users are reluctant to sign up to a service precisely because they don't want to surrender their data because once it's gone it's gone forever. I suspect users will be more open to trying new services if they can be assured that it's possible to "un-sign up", that is be "forgotten" by a service.
- Steeeve 9y ago
  Does the GDPR actually identify what information is considered personal? I haven't followed it at all. For instance, if someone posted on a message board, is it enough to rename their user to anonymous. Or do you have to go back and delete their user, leaving orphaned records? Or do you have to delete all of their postings, which could leave discussion history in disarray. What about something like a phone service? Erasing a lines recent history is easy enough, but going back years to delete records from archival systems that weren't designed to handle it could be problematic. For instance, in very large data tables, deletes can be very slow. Call records are often stored in compressed flat files. Which would mean searching through tens of thousands flat files for lines to delete. And some of that data would have been processed through a system like splunk or logstash that isn't particularly friendly to deletes and would require a massive re-indexing operation to flush the necessary records. And some of those systems probably have tiered storage that includes offline, slow to recover archives built with cost assessments that did not account for frequent data removal. (Think about how much it would cost to download 500TB from glacier, decompress it, decrypt it, put it on an active system, remove a few records, re-encrypt, re-compress and re-upload it 4 or 5 times a month). And think of how that cost compares with "we generally need to reference maybe 1gb of data every three or four months".
ivan_gammel 9y ago
One important thing not to forget about: make sure your event log doesn't leak metadata. You may have deleted PII like email or username, but if certain user activity logged under some user id provides sufficient information to identify the user (e.g. a friend tags user on his group photo - this event contains just an id, but allows to link specific person on still available image to that id), what will you do?
- arpa 9y ago
  That is an interesting point - a photo by itself is personally identifiable information[1] - so you should blur/distort/remove that particular face - or, to stay on the safe side - all faces, just like google maps does to be GDPR compliant, i guess. Jesus, this regulation is such a can of worms... [1] https://www.quora.com/Is-a-photo-of-a-person-considered-PII-personally-identifiable-information-without-metadata-like-their-name https://www.quora.com/Is-a-photo-of-a-person-considered-PII-...
  - ivan_gammel 9y ago
    This logic would effectively kill all photo-sharing services. I guess, noone wants to blur people's faces in their Instagram posts,so a regulation following the old Islamic hadith not to depict humans is really terrible perspective.
  - TheForumTroll 9y ago
    If you upload personal identifiable information about me surely I should have the right to get it removed? You already have no right to upload the data in the first place without my explicit permission, so the problem in your scenario is not GDPR compliance unless I gave you permission and if I did I have the right to remove the permission at any point in time already if I so chose. I don't see how GDPR changes anything in that scenario.
    - Geee 9y ago
      The user who posts the photo is a data controller, and it is her obligation under GDPR to remove the photo when requested. All data processors in the chain (the photo app, cache, host, etc.) have to provide the means for the data controller to remove the image. That's my understanding for user-generated content which may or may not contain personal information.

by arpa 9y ago

jshen 9y ago

The whole point of event sourcing is the ability to manage state effectively. When you move state out of the event sourcing model you are losing the state management benefits of event sourcing for that data. Maybe that’s a reasonable trade-off for GDPR, but there is a trae-off.

ithkuil 9y ago
There are ways to put some state outside the event stream that play nicely with the model. For example you could store the body in a content addressed store (a key value store whose keys are hashes of the value). This preserves the ideal model of immutable events as you cannot accidentally change the content of the event since that would violate the property of the CAS. But nothing prevents you from deleting an entry in the CAS. However you still have to find all the records you need to delete! The trick with forgetting the encryption key let makes this step a O(1) operation. On the other hand it forces you to decide upfront the granularity of ownership, i.e. if you treat payload deletion as a batch process you can deal with mistakes, change your mind about which data should or should not be deleted. While with the encryption technique, changing your mind would require you to perform a full history replay and reencrypt data in a new stream (and throw away the old one) with the new granularity model.

michaelbuckbee 9y ago

This is actually a recommended approach within the GDPR legislation itself Article 25: "Pseudonymization" https://blog.varonis.com/gdpr-requirements-list-in-plain-english/#article25 https://blog.varonis.com/gdpr-requirements-list-in-plain-eng...

dnomad 9y ago

I think that's exactly what any real system would do. This sort of widespread event encryption and forgetting is crazy. It is a very bad idea to delete events from the immutable event log. There is no telling how downstream consumers will react but it probably won't be good. It's much simpler to only hold references to a user (UserID) and then if you need to forget that a user you simply burn whatever is behind that UserID. This is actually how the GDPR is designed: it's not about user data it's about personal data, that is data that can be associated or related back to an "identifiable person." The problem is not that you need to delete all the user's data across all your systems, the problem is that you need to break any associations that would allow you to identify a natural person who has asked to be "forgotten." The funny thing is that while some people make lots of noises about GDPR being some huge "burden," the reality is that any architect worth her salt should've been designing systems like this from the very start rather than letting personal data be replicated en mass from one system to another. It is a basic normalization of data. All the GDPR is doing, like most regulations, is requiring businesses follow best practices and not cut corners that might harm end users. The great thing about this in the long run is that this fixes a huge problem with the internet. Today users are reluctant to sign up to a service precisely because they don't want to surrender their data because once it's gone it's gone forever. I suspect users will be more open to trying new services if they can be assured that it's possible to "un-sign up", that is be "forgotten" by a service.

Steeeve 9y ago
Does the GDPR actually identify what information is considered personal? I haven't followed it at all. For instance, if someone posted on a message board, is it enough to rename their user to anonymous. Or do you have to go back and delete their user, leaving orphaned records? Or do you have to delete all of their postings, which could leave discussion history in disarray. What about something like a phone service? Erasing a lines recent history is easy enough, but going back years to delete records from archival systems that weren't designed to handle it could be problematic. For instance, in very large data tables, deletes can be very slow. Call records are often stored in compressed flat files. Which would mean searching through tens of thousands flat files for lines to delete. And some of that data would have been processed through a system like splunk or logstash that isn't particularly friendly to deletes and would require a massive re-indexing operation to flush the necessary records. And some of those systems probably have tiered storage that includes offline, slow to recover archives built with cost assessments that did not account for frequent data removal. (Think about how much it would cost to download 500TB from glacier, decompress it, decrypt it, put it on an active system, remove a few records, re-encrypt, re-compress and re-upload it 4 or 5 times a month). And think of how that cost compares with "we generally need to reference maybe 1gb of data every three or four months".

ivan_gammel 9y ago

One important thing not to forget about: make sure your event log doesn't leak metadata. You may have deleted PII like email or username, but if certain user activity logged under some user id provides sufficient information to identify the user (e.g. a friend tags user on his group photo - this event contains just an id, but allows to link specific person on still available image to that id), what will you do?

arpa 9y ago
That is an interesting point - a photo by itself is personally identifiable information[1] - so you should blur/distort/remove that particular face - or, to stay on the safe side - all faces, just like google maps does to be GDPR compliant, i guess. Jesus, this regulation is such a can of worms... [1] https://www.quora.com/Is-a-photo-of-a-person-considered-PII-personally-identifiable-information-without-metadata-like-their-name https://www.quora.com/Is-a-photo-of-a-person-considered-PII-...
- ivan_gammel 9y ago
  This logic would effectively kill all photo-sharing services. I guess, noone wants to blur people's faces in their Instagram posts,so a regulation following the old Islamic hadith not to depict humans is really terrible perspective.
- TheForumTroll 9y ago
  If you upload personal identifiable information about me surely I should have the right to get it removed? You already have no right to upload the data in the first place without my explicit permission, so the problem in your scenario is not GDPR compliance unless I gave you permission and if I did I have the right to remove the permission at any point in time already if I so chose. I don't see how GDPR changes anything in that scenario.
  - Geee 9y ago
    The user who posts the photo is a data controller, and it is her obligation under GDPR to remove the photo when requested. All data processors in the chain (the photo app, cache, host, etc.) have to provide the means for the data controller to remove the image. That's my understanding for user-generated content which may or may not contain personal information.