5 ms·

I think that's exactly what any real system would do. This sort of widespread event encryption and forgetting is crazy. It is a very bad idea to delete events f

by dnomad 9y ago

I think that's exactly what any real system would do. This sort of widespread event encryption and forgetting is crazy. It is a very bad idea to delete events from the immutable event log. There is no telling how downstream consumers will react but it probably won't be good. It's much simpler to only hold references to a user (UserID) and then if you need to forget that a user you simply burn whatever is behind that UserID. This is actually how the GDPR is designed: it's not about user data it's about personal data, that is data that can be associated or related back to an "identifiable person." The problem is not that you need to delete all the user's data across all your systems, the problem is that you need to break any associations that would allow you to identify a natural person who has asked to be "forgotten." The funny thing is that while some people make lots of noises about GDPR being some huge "burden," the reality is that any architect worth her salt should've been designing systems like this from the very start rather than letting personal data be replicated en mass from one system to another. It is a basic normalization of data. All the GDPR is doing, like most regulations, is requiring businesses follow best practices and not cut corners that might harm end users. The great thing about this in the long run is that this fixes a huge problem with the internet. Today users are reluctant to sign up to a service precisely because they don't want to surrender their data because once it's gone it's gone forever. I suspect users will be more open to trying new services if they can be assured that it's possible to "un-sign up", that is be "forgotten" by a service.

Steeeve 9y ago
Does the GDPR actually identify what information is considered personal? I haven't followed it at all. For instance, if someone posted on a message board, is it enough to rename their user to anonymous. Or do you have to go back and delete their user, leaving orphaned records? Or do you have to delete all of their postings, which could leave discussion history in disarray. What about something like a phone service? Erasing a lines recent history is easy enough, but going back years to delete records from archival systems that weren't designed to handle it could be problematic. For instance, in very large data tables, deletes can be very slow. Call records are often stored in compressed flat files. Which would mean searching through tens of thousands flat files for lines to delete. And some of that data would have been processed through a system like splunk or logstash that isn't particularly friendly to deletes and would require a massive re-indexing operation to flush the necessary records. And some of those systems probably have tiered storage that includes offline, slow to recover archives built with cost assessments that did not account for frequent data removal. (Think about how much it would cost to download 500TB from glacier, decompress it, decrypt it, put it on an active system, remove a few records, re-encrypt, re-compress and re-upload it 4 or 5 times a month). And think of how that cost compares with "we generally need to reference maybe 1gb of data every three or four months".
- Radim 9y ago
  The law is not defined in a "technical" way. It's definitely not about renaming users or deleting their postings. GDPR is declarative: can you identify someone through the records in your possession, whether database or not, and connect that to any protected information? (their political views, sexual preferences, name and location, economic status, health issues…). The law is quite broad and fuzzy, which is what makes technical people uneasy: ‘personal data’ means any information relating to an identified or identifiable natural person (‘data subject’); an identifiable natural person is one who can be identified, directly or indirectly, in particular by reference to an identifier such as a name, an identification number, location data, an online identifier or to one or more factors specific to the physical, physiological, genetic, mental, economic, cultural or social identity of that natural person; It is entirely possible that in your message board example, you've deleted the user, you've deleted their postings, and are still in violation. For one, the person requesting deletion doesn't even have to be your user. Someone else may have posted such information about them on your board. People misconstrue GDPR to be only about databases and unlinking users' records from tables (mainly because it's the easiest thing to do). But it really is about all and any personal and sensitive information of natural people, full stop.
  - Geee 9y ago
    In that case, individuals are data controllers and not data processors. Everyone posting on an online forum controls the data they post. Adding 'delete post' button fulfils the requirement for a data processor.
  - aeorgnoieang 9y ago
    > For one, the person requesting deletion doesn't even have to be your user. If what you wrote is true, then organizations are liable for data that someone may, possibly even just theoretically, be able to use to identify someone else. Given that the law is not 'technical', maybe it'll be interpreted much more leniently than a straightforward reading would lead one to expect.
    - Radim 9y ago
kingofhdds 9y ago
GDPR is troublesome not for requiring something particular which is hard to implement, but because it requires to do something poorly defined, and promising heavy fines for not following the path one can't clearly see. This minefield situation, in fact, makes SMEs much more vulnerable compared to big corporations, and I suppose it's not what most people here would like to see.
Terr_ 9y ago
> if you need to forget that a user you simply burn whatever is behind that UserID [...] you need to break any associations that would allow you to identify a natural person The problem is that as you collect more "impersonal" data, the probability that your collective data can still be used to identify someone approaches 100%. "We don't know their name, but Deleted User 4510 was friends with every single member of the John Doe family except for John Doe himself..."
- joosters 9y ago
  Surely the implication is that when you 'burn whatever is behind that UserID' you would delete the records of who DeletedUser4510 was friends with (and who was friends with DeletedUser4510). If all you do is rename someone to DeletedUser4510, you pretty obviously haven't deleted all the data you hold on them.
  - throwawayReply 9y ago
    Right, and what if those connections are burned into an immutable data source through the connection event in your event sourcing? That is what this article is about, and methods to deal with that.
  - Terr_ 9y ago
    > you would delete the records of who DeletedUser4510 was friends with You erase the friend-linkages... ... Then you find that Jane Doe made a post in response to a blanked DeletedUser4510 post, and responses can only be made by friends, so therefore Jane was at some point friends with DeletedUser4510. So you put in tombstones for all posts and all post-to-post causation links. ... Then you find that the entire Doe family is tagged in a photo by Jebediah Doe, except for one guy, and the comments are every family member and somebody named DeletedUser4510... Anyway, my point is that it's really easy to get caught in such a fog of relational data that gaps merely change a certainty of identification down to an extremely high probability, and an event-sourced system -- by design -- makes it extra difficult to remove data or to even plan for its removal.
codetrotter 9y ago
> It's much simpler to only hold references to a user (UserID) and then if you need to forget that a user you simply burn whatever is behind that UserID. But if someone was a user of your service and your services included say photo or video hosting, and you delete their name, their phone number and such, but all of the data you don’t delete still reference a common user ID, then that all of your data can still be related back to an identifiable person. If instead of holding reference to the user id on the data themselves you have separate table for data ownership that is one step better. I.e you have all data stored separately and each piece of data has a guid, and you have the user profile separate with a guid and then you have separate tables tying data guid to user guid and when you delete a profile you delete both the user profile and the tables that tie data to that user profile guid. However I think even that is not enough. Yes it’s hard to implement full deletion, but you are the one that chose to accept data in the first place. If you can’t implement a system where the data can be deleted you shouldn’t be accepting that data in the first place IMO. Because even if you delete the references to the data, the individual pieces of data can still be tied back together with data analysis. For example by looking at meta data in pictures, or looking for artifacts that come from lens scratches. Also, even if you delete references of ownership of an image, probably you aren’t deleting records of comments that other people made on the photo, because if you went to that extent you probably could have just deleted the data proper, so then with access to the image and knowing what other people commented it will likely be possible to determine who originally uploaded that photo.
- arpa 9y ago
  Not to derail, but rather share: I have worked on a healthcare solution which did just that, every medical record had a guid, and by itself it didn't carry a direct relation to a patient; it was done via an additional table. This was done to ensure data security, ability to use the medical records for scientific purposes (records themselves were anonymous) and to manage access to these records (i.e. patient was the owner of his/her records by default, but could choose to delegate read and modification permissions to e.g. their spouse or physician, or emergency personnel). Come to think of it, it was a pretty awesome system, even though it ran on asp.net (it was some 7 years ago i think). This design was necessary because data privacy regulations for medical records have always been very strict in the EU; come to think of it, people worried about GDPR should take a look at the way medical systems handle data.

by dnomad 9y ago

Steeeve 9y ago

Does the GDPR actually identify what information is considered personal? I haven't followed it at all. For instance, if someone posted on a message board, is it enough to rename their user to anonymous. Or do you have to go back and delete their user, leaving orphaned records? Or do you have to delete all of their postings, which could leave discussion history in disarray. What about something like a phone service? Erasing a lines recent history is easy enough, but going back years to delete records from archival systems that weren't designed to handle it could be problematic. For instance, in very large data tables, deletes can be very slow. Call records are often stored in compressed flat files. Which would mean searching through tens of thousands flat files for lines to delete. And some of that data would have been processed through a system like splunk or logstash that isn't particularly friendly to deletes and would require a massive re-indexing operation to flush the necessary records. And some of those systems probably have tiered storage that includes offline, slow to recover archives built with cost assessments that did not account for frequent data removal. (Think about how much it would cost to download 500TB from glacier, decompress it, decrypt it, put it on an active system, remove a few records, re-encrypt, re-compress and re-upload it 4 or 5 times a month). And think of how that cost compares with "we generally need to reference maybe 1gb of data every three or four months".

Radim 9y ago
The law is not defined in a "technical" way. It's definitely not about renaming users or deleting their postings. GDPR is declarative: can you identify someone through the records in your possession, whether database or not, and connect that to any protected information? (their political views, sexual preferences, name and location, economic status, health issues…). The law is quite broad and fuzzy, which is what makes technical people uneasy: ‘personal data’ means any information relating to an identified or identifiable natural person (‘data subject’); an identifiable natural person is one who can be identified, directly or indirectly, in particular by reference to an identifier such as a name, an identification number, location data, an online identifier or to one or more factors specific to the physical, physiological, genetic, mental, economic, cultural or social identity of that natural person; It is entirely possible that in your message board example, you've deleted the user, you've deleted their postings, and are still in violation. For one, the person requesting deletion doesn't even have to be your user. Someone else may have posted such information about them on your board. People misconstrue GDPR to be only about databases and unlinking users' records from tables (mainly because it's the easiest thing to do). But it really is about all and any personal and sensitive information of natural people, full stop.
- Geee 9y ago
  In that case, individuals are data controllers and not data processors. Everyone posting on an online forum controls the data they post. Adding 'delete post' button fulfils the requirement for a data processor.
- aeorgnoieang 9y ago
  > For one, the person requesting deletion doesn't even have to be your user. If what you wrote is true, then organizations are liable for data that someone may, possibly even just theoretically, be able to use to identify someone else. Given that the law is not 'technical', maybe it'll be interpreted much more leniently than a straightforward reading would lead one to expect.
  - Radim 9y ago

kingofhdds 9y ago

GDPR is troublesome not for requiring something particular which is hard to implement, but because it requires to do something poorly defined, and promising heavy fines for not following the path one can't clearly see. This minefield situation, in fact, makes SMEs much more vulnerable compared to big corporations, and I suppose it's not what most people here would like to see.

Terr_ 9y ago

> if you need to forget that a user you simply burn whatever is behind that UserID [...] you need to break any associations that would allow you to identify a natural person The problem is that as you collect more "impersonal" data, the probability that your collective data can still be used to identify someone approaches 100%. "We don't know their name, but Deleted User 4510 was friends with every single member of the John Doe family except for John Doe himself..."

joosters 9y ago
Surely the implication is that when you 'burn whatever is behind that UserID' you would delete the records of who DeletedUser4510 was friends with (and who was friends with DeletedUser4510). If all you do is rename someone to DeletedUser4510, you pretty obviously haven't deleted all the data you hold on them.
- throwawayReply 9y ago
  Right, and what if those connections are burned into an immutable data source through the connection event in your event sourcing? That is what this article is about, and methods to deal with that.
- Terr_ 9y ago
  > you would delete the records of who DeletedUser4510 was friends with You erase the friend-linkages... ... Then you find that Jane Doe made a post in response to a blanked DeletedUser4510 post, and responses can only be made by friends, so therefore Jane was at some point friends with DeletedUser4510. So you put in tombstones for all posts and all post-to-post causation links. ... Then you find that the entire Doe family is tagged in a photo by Jebediah Doe, except for one guy, and the comments are every family member and somebody named DeletedUser4510... Anyway, my point is that it's really easy to get caught in such a fog of relational data that gaps merely change a certainty of identification down to an extremely high probability, and an event-sourced system -- by design -- makes it extra difficult to remove data or to even plan for its removal.

codetrotter 9y ago

> It's much simpler to only hold references to a user (UserID) and then if you need to forget that a user you simply burn whatever is behind that UserID. But if someone was a user of your service and your services included say photo or video hosting, and you delete their name, their phone number and such, but all of the data you don’t delete still reference a common user ID, then that all of your data can still be related back to an identifiable person. If instead of holding reference to the user id on the data themselves you have separate table for data ownership that is one step better. I.e you have all data stored separately and each piece of data has a guid, and you have the user profile separate with a guid and then you have separate tables tying data guid to user guid and when you delete a profile you delete both the user profile and the tables that tie data to that user profile guid. However I think even that is not enough. Yes it’s hard to implement full deletion, but you are the one that chose to accept data in the first place. If you can’t implement a system where the data can be deleted you shouldn’t be accepting that data in the first place IMO. Because even if you delete the references to the data, the individual pieces of data can still be tied back together with data analysis. For example by looking at meta data in pictures, or looking for artifacts that come from lens scratches. Also, even if you delete references of ownership of an image, probably you aren’t deleting records of comments that other people made on the photo, because if you went to that extent you probably could have just deleted the data proper, so then with access to the image and knowing what other people commented it will likely be possible to determine who originally uploaded that photo.

arpa 9y ago
Not to derail, but rather share: I have worked on a healthcare solution which did just that, every medical record had a guid, and by itself it didn't carry a direct relation to a patient; it was done via an additional table. This was done to ensure data security, ability to use the medical records for scientific purposes (records themselves were anonymous) and to manage access to these records (i.e. patient was the owner of his/her records by default, but could choose to delegate read and modification permissions to e.g. their spouse or physician, or emergency personnel). Come to think of it, it was a pretty awesome system, even though it ran on asp.net (it was some 7 years ago i think). This design was necessary because data privacy regulations for medical records have always been very strict in the EU; come to think of it, people worried about GDPR should take a look at the way medical systems handle data.