3 ms·
Request to purge data are one aspect of the GDPR that Pachyderm makes trickier. It makes it easier to remove a piece of data and recompute all of your models wi
by jdoliner 7y ago
Request to purge data are one aspect of the GDPR that Pachyderm makes trickier. It makes it easier to remove a piece of data and recompute all of your models without it, because it can deduplicate the computation. But to truly purge a piece of data deduplication becomes a hinderance, because the data can be reference by previous commits, and even by other user's data. You can delete a piece of data and have it not be truly purged.
The best recommendation we have for that is that user's data should be encrypted with a key that's unique to the user, and when that user asks you to purge their data you should throw away the key. That means that even if two users have the same data it will be stored encrypted by different keys, so if one asks for the data to be purged the other can still keep their data.
- visarga 7y agoBut then wouldn't the storage and distribution of keys become a similar problem to the original one? If the keys get distributed, then it's hard to really remove them.
- jdoliner 7y agoYes, all the keys do is scale the problem down. In general this is a very tough problem, everything else in the system is designed to avoid data loss, that's the biggest scariest failure case. But then when you want to lose data all the measures in the system to prevent data loss prevent that from happening.