7 ms·

How to balance between delete compliance and backups

69 points by nikanj 5y ago

[ Inspired by Atlassians outage ( https://www.atlassian.com/engineering/april-2022-outage-update ) ] Dear HN, What are the industry best practices for data deletion? Compliance rules stipulate that data can be permanently deleted when necessary, but backup policies stipulate that data can be restored if needed. In this case, Atlassian ran what they called the "permanently delete" capability that is required to permanently remove data when required for compliance reasons. They quickly learned that they deleted the wrong data, and started the process of restoring it from their backups. How do various regulators view this? A layman interpretation says that the data was not permanently deleted, because it was still in the backups. How to balance the two requirements?

mhitza 5y ago
As long as your end users are well informed about your backup retention policy, and when restoring from those backups you make sure to delete the user data; I see no issue.
paulsutter 5y ago
Back up encrypted data and discard keys selectively to delete data from the archive
- bottled_poe 5y ago
  This is an excellent solution. I don’t see why it couldn’t be the industry standard.
  - Silhouette 5y ago
    It's a solution that looks clever at first sight but the devil is in the details. Sometimes it's difficult to determine what data is subject to erasure in the first place, for example if there are multiple parties involved or the obligation to erase conflicts with other obligations that might require retention of some of the personal data. If you're going with the encryption and key management strategy, your design must accommodate all possible scenarios both now and in the future including those where the rules change between now and then. For anything but the simplest cases that seems unlikely.
    - bottled_poe 5y ago
      Seems easy to me. 30 days, delete backup keys. What’s the problem? You still have 29 day old backups?
      - Silhouette 5y ago
        Maybe we're talking about slightly different things here. The suggestion to use individual keys to encrypt personal data is usually made so that deleting a key has effectively and immediately destroyed all copies of the data including any historical ones. That way you're definitely fully compliant with any right to be forgotten requests with one simple act. Even if you do have some older backups or archives kept around or the backup data isn't truly irretrievable from the physical media where it was stored after being marked for deletion, that data instantly becomes random noise without the key (which you obviously don't store and backup in the same way in this kind of scheme). In other words, it's not really about whether keeping backups for a few days or a few weeks before they are "deleted" would be considered acceptable by regulators, it's about making all such questions irrelevant while otherwise being able to continue with business as usual. It would be a clever and useful solution to the regulatory compliance problem in this area, if it worked in practice.
  - jandrewrogers 5y ago
    It doesn't work in real systems because it is extremely costly to the point of being infeasible. This has been known for decades and rejected because it doesn't actually work. If a technical solution is "obvious" and no one is using it, it tends to imply that it has serious flaws that are obvious to domain experts. Using encryption for deletion is known to be deeply flawed for unavoidable technical reasons. These databases actually exist. Almost no one uses them because the performance is exceptionally poor to the point of being nearly unusable for practical things.
    - bottled_poe 5y ago
      I disagree. Why doesn’t time-bounded deletion of backup keys work?
- dogma1138 5y ago
  That doesn’t scale, if you need to delete records pertaining to a specific person and you have a user base of 10M yet alone 100M or 2B users you can’t encrypt each of their records with an individual key.
  - bottled_poe 5y ago
    I think the parent is suggesting discarding keys for all backups after X days. So, anything deleted in the live environment effectively gets removed from the backups after X days.
  - lucb1e 5y ago
    Depends on how you back up. If you copy the raw database files that you also run queries on (e.g. take a zfs snapshot), then you'd need some very special encryption to be able to query across differently-encrypted data. But if your database is just on an encrypted disk but otherwise readable, and you only encrypt the records on an individual level when exporting them to the backup system... that can work. Generating a secure random encryption key after seeding can be as fast as a single AES function call, and symmetric encryption isn't particularly expensive either. Still extra work compared to just copying bytes between memory regions (now everything has to be operated on by the CPU), I'm not saying it's going to be trivial, but it doesn't seem infeasible. Also at 2B users as you mention, which is an extreme example considering we have only a few billion internet users (multiply by one or two bots for every person, but even then), you need huge storage and compute clusters anyway. Storing 16 bytes of key material extra isn't going to be the difference between feasible and infeasible. It doesn't really matter if you're talking 2e2 or 2e9 users in terms of computational or storage overhead.
  - benlivengood 5y ago
    Large companies actually do keep individual master keys per user. Think of it this way, what's an extra 256 bits (32 bytes) stored for every user in a system when their name and email address probably exceeds 32 bytes already? Master keys encrypt the records of a user-key database where user keys are stored. User keys protect unique wrapped keys that are randomly generated at time of encryption (perhaps this is where your logistical argument arose) for individual pieces of data. This is commonly implemented as an encryption service that will encrypt/decrypt blobs of data for services on behalf of a particular user via RPC; the cryptographic keys are never present in normal service code or standard servers and only within the encryption service (which may be an HSA) so there's a much smaller domain to secure. Common ACLs protect calls to the encryption service per user, like they protect access to storage and other resources. Obviously for bulk data encryption this is less tenable so services handling bulk data (many GB) get extra protection and direct access to unwrapped user keys, but must be responsible for respecting privacy/security ACLs internally.
- contravariant 5y ago
  This might be appropriate for some cases, but is dangerously similar to just not making backups in the first place.
- SnowHill9902 5y ago
  If I use a 16 bit key does it count as deleted data?
  - lucb1e 5y ago
    Roughly equivalent to hashing a small-ish number like a phone numbers, so based on [1] I'd say yes! Marketing can now say with a straight face that through this technical operation, basically sprinkling some magic pixie dust over your data, it is now incomprehensible and no longer leads back to you. [1] 7:05 - 7:58 https://projectbullrun.org/surveillance/2015/video-2015.html#bernstein https://projectbullrun.org/surveillance/2015/video-2015.html... (2015)
- twhitmore 5y ago
  It just moves the problem, to how do you delete the keys & manage backups of the keys. You have to backup the keys, since a system outage would otherwise leave you unable to function. My shot from the hip: encryption/keying is irrelevant except for highly sensitive data (credit cards, crypto wallets). Plain deletion from production databases should be sufficient, backups should be "at arms length" but otherwise should not need to be treated specially. Another shot from the hip: System outage & RPO loss of a requested deletion is an act of God and no worse (indeed probably less important) than losing the other business transactions in that time period. I don't see special handling being needed for deletion.
lucb1e 5y ago
This seems like a false dichotomy or, at best, a very vague discussion if you don't point out a specific regulation to run afoul of. Taking GDPR and its ~1995 predecessor as an example, which both required personal data to be collected only when there was a reason for it, thus when you have no more use for it, it has to be deleted. However, backups are a perfectly legit reason. It's not an excuse to keep data into eternity, but checking that all systems are good after a delete run, or having a buffer period of some defined time, is definitely reasonable and should (not a lawyer) not run afoul of the regulation. So I am not sure what exactly it is you are concerned about.
Terry_Roll 5y ago
>How do various regulators view this? Its a good question, one I've mentioned on YV in the past. I havent seen it handled in the UK, but the backup industry isnt handling it either from what I've seen. I think the regulators will view it as it needs to deleted from backups, but the backup software industry doesnt really make it easy to delete stuff from said backups. It could be done because individual file & mailbox restores exist so if they can identify what needs to be restored, they could just as easily delete it.
macintux 5y ago
Title should be prefaced with “Ask HN:”
colonelxc 5y ago
It's generally not required to 'permanently delete instantly' (this includes GDPR). So you take a backup every day, and delete backups after 30 days (or whatever amount of time you need to for compliance). When somebody says 'delete this permanently', you do, from your primary data store. Then after 30 days, not even the backups will have anything. You can also get fancy with more than one stage (soft delete, hard delete, then backup purge), as long as the whole pipeline is done within the compliance window. Of course, the trickiest part of all of this is knowing where all data can proliferate, and making sure it doesn't (don't let people download data to their machines, don't let people create a 'copy of prod' that sticks around outside or retention.
phire 5y ago
I'm of the opinion that as long as a company has policies to never restore legitimately deleted data from backups, then backups of deleted data simply shouldn't count. Everything else is just too ugly to implement. Though I get the impression that regulators are not happy with this opinion, and what that data to be physically deleted within some timeline.
Silhouette 5y ago
In the UK, the regulator has a specific comment on this exact issue: https://ico.org.uk/for-organisations/guide-to-data-protection/guide-to-the-general-data-protection-regulation-gdpr/individual-rights/right-to-erasure/#ib5 https://ico.org.uk/for-organisations/guide-to-data-protectio...
netman21 5y ago
A technical reading of GDPR leads one to think that data in a backup should also be deleted. But in practice most orgs are treating Article 7, The Right to Erasure, as meaning "make no longer publicly available." That still leaves the issue of ensuring processes are in place to make sure the "deleted" data is not restored from backups.
notreallyserio 5y ago
Oof, only now do I realize I need to record account deletions somewhere independent of my database backups so I don't accidentally restore accounts I shouldn't if something goes horribly wrong. Not sure why I didn't think of this before.
- kingcharles 5y ago
  Ah, this does add a layer of complexity I hadn't thought of before.
Nextgrid 5y ago
Legitimate interest (as mentioned in the GDPR) is your friend here. GDPR enforcement is not black and white and there is no specific rules or timeframes you’re allowed to keep data - instead, if there is a complaint, the regulator will work with you to understand your argument before any penalties are handed out, so you’ll be fine unless you’re obviously acting in bad faith. So deleting data from the primary data store while keeping backups for a reasonable amount of time (3 months?) is totally fine even though you are technically right in that the data hasn’t yet been fully deleted.
tedk-42 5y ago
A 'Mark for deletion' coupled with a check once the deletion marker should be processed has succeeded would be a way to avert what happened at Atlassian. No matter how well built a system, human error can still happen which leads us down dark paths. Also, no backup currently exists for when an alien race blows up the Earth. Some things are simply impossible to account for.
zinodaur 5y ago
A pattern that has saved my bacon a number of times: When writing the "permanent delete" code, only issue the delete when you have proven that there is a backup. i.e.: - Create a backup of the data you want to delete and store it somewhere - Read everything in the backup, and delete everything you find in the backup from prod - Have the backup expire after a certain time threshold (e.g. s3 has a nice TTL setting for backup buckets) Even if you delete the wrong thing, the backups are always immediately accessible in the near-term
- ledgerdev 5y ago
  I like this, excellent way to handle permanent record deletion.
tgsovlerkhgsel 5y ago
First of all, most privacy regulations are surprisingly toothless and rarely enforced, so in practice, you get away with anything until you somehow do draw the ire of a regulator, and then they'll enforce their random favorite pet rules that don't make much sense, and may also change arbitrarily. (Think "must use fax, can't use e-mail unless it's PGP-encrypted" followed by "can't use fax either".) German regulators that I spoke with acknowledge the need for backup, but require you to keep a separate "deletion log", so that when data is restored from backup, you can replay the deletions and keep supposed-to-be-deleted data from entering live systems. You also can't keep the backups forever, but if you have a reasonable plan, you're most likely going to be OK (even if they disagree, they'll probably just tell you to change your retention periods). This applies to privacy in general - as long as you show a minimum of genuine care, and deal with a few formal requirements, you're probably going to be OK. GDPR is most scary for people who a) think it will be enforced to the fullest extent that a pessimistic reading of the law might allow, or b) are actively doing privacy-hostile things that they know they shouldn't be doing but want to keep doing. For example, storing the data for backup purposes, then going back in and using it to build ML models because the backups have more data than the live system would (rightfully) have a decent chance of getting you in trouble. You can keep data for backup purposes, but if you're allowed to store data for purpose A, that doesn't mean "I'm allowed to store it so now I can use it for any purpose" as some people seem to think.
ledgerdev 5y ago
> What are the industry best practices for data deletion? Tokenization. Exchange the data for tokens, and store the sensitive data someplace very secure separate from the main operational system. To process a delete, delete the token. Ideally this is handled via proxy, so it's a completely separate system from primary app. Having used this method, I feel it ought to be used for every app storing sensitive data. edit: I forgot to mention the biggest issue I have with crypto shredding (encryption per record) is that you can't correlate data between records as you can with tokens.
- kevin_nisbet 5y ago
  It's not clear to me that this is really sufficient, or even really answers the original question. On the sufficient side, I don't know that retaining an encrypted version of customer data would really meet a lot of compliance definitions for deleted data. Atleast, nothing I've heard of tested in court. Also, depending on the data, even tokenized / encrypted / etc approaches are easy to get wrong and still allow some correlation. In a previous company when it came to network tooling, we had to be extremely careful that we didn't intercept 1 bit of payload, as even though that 1 bit couldn't be used to meaningfully reproduce a phone call, by many legal definitions we would be illegally intercepting phone calls. And then circling back to the original question, the problem of how to manage data and customer deletions if just shifted to how to do this with the tokens. So my take on the original question is as follows. To my knowledge, the right to be forgotten compliance regimes do include time windows for deletions, understanding that it can take time to purge requested data. Also, some customers may want this in their contracts, so I'd try and align any contracts / terms of service with the most strict compliance regimes. When taking backups, if possible use WORM (on S3 I think this is called object locks), and set the lock to be less time than the compliance regime requires, but long enough that there is always locked backups that can't be deleted by a mistake or adversary. Depending on required history, you can then just let the backups expire. If you need longer backup history than the compliance regimes allow, then it comes to structuring backups so selective data can be removed, and then relocking the backup. And doing this on a rolling basis, so at no point in time or run of the cleanup are all backups being touched at the same time.
- TazeTSchnitzel 5y ago
dasil003 5y ago
You don’t. Restoring a backup is rare. If this happens to restore a deleted data, the deletion request is part of the lost data. Trying to work around this with a separate delete request log doesn’t solve the problem, it just shrinks the loophole. But if you are restoring often enough for this to draw the ire of regulators you probably have much bigger problems. Of course I’m just giving you the technical truth straight up. The other part of this is you would never say it so directly to a regulator or auditor. Work with your compliance team to get the messaging right.
- zeckalpha 5y ago
  Unless they are different databases.
swiftcoder 5y ago
The way most bigger companies deal with this is by making all deletions soft-deletes (i.e. just set a deleted=true flag in the database record), and then scheduling the hard deletion as a cancellable async process to be run some time in the future (say, 90 days). It's very hard to cause massive outages with soft deletes, it's easy to rollback, and compliance rules rarely put a tight deadline on mandatory deletions (i.e. that 90 day deletion window is GDPR compliant)
jasonkester 5y ago
My approach involves two steps: 1. Never permanently delete anything. 2. Never implement anything that makes your product worse because of some silly regulation. So if you “delete” your cat picture from one of my sites, I’ll flip the IsActive bit to false for you and nobody will ever see it again. Then when you realize that it was your favorite cat picture and how could you delete it forever I never told you to do that I’ll sue you, I can flip that IsActive bit back to true. That situation has happened way more often than the one where the New Zealand Data Storage Task Force Enforcement Division has levied fines against me. So it’s the best strategy.
- furstenheim 5y ago
  GDPR fines are pretty hefty. I doubt the employer would be happy with soft deletion. Also, you don't need an auditor. In the eventuality of a hack, the db could be made open and then it would become clear you were not deleting
- TazeTSchnitzel 5y ago
  This is a very bad idea for GDPR compliance, but it is an even worse idea if you don't want to be charged with possession of CSAM.
traceroute66 5y ago
Forgive me, but I'm not sure why this is even a question. Compliance comes first, because if you don't comply with legislation you most likely won't be in business for long. So you then build on from that by designing your system to take compliance aspects into account. If you do that, then the backup question no longer occurs, because your system design has taken into account the fact that (for example) GDPR "Right to be forgotten" exists, and so you will have ensured you have a mechanism in place to ensure the "forgotten" PII does not make its way into your backups.
ealexhudson 5y ago
Backing up data that is later permenantly deleted is fine. From a regulatory point of view, typically you only need to ensure: * the backups are not "within reach" and that there's no danger of data that has been deleted being looked up/used. * any restoration of backups will not result in deleted data coming back to life. This is the tougher requirement, and means you do need to be able to re-delete things somehow - but typically this is only an issue in scenarios where you're having to undo damage that went undetected for a long time, so you have other big fish to fry also. There are often competing compliance concerns here. E.g. a legal hold on data should prevent stuff being deleted, and in some instances may allow you to go back into backups to retrieve deleted things.
- csydas 5y ago
  I want to second and elaborate on the above statement. As I work heavily with backups, this is something that is fraught with a lot of arm-chair analyses, which I will try to avoid, and instead focus on just how the industry has reacted. For Backups, most vendors have two takes on this, and my hot take is that it's less about legal interpretations and it's how the vendors spin their backup technology features: 1. Vendors that have mutable backups (e.g., the backup archive content can be changed and individual records, file, and other stuff can be edited/removed) finally have a defense against the security risk of mutable backups and now it's a compliance feature 2. Vendors with non-mutable backups (e.g., the backup file itself is still mutable (e.g., no immutable bit), but you cannot remove individual records/entries from it without destroying the backup file itself) have simply developed staging restores that allow for programatic removal of a non-mutable restored machine before performing an actual restore. The parent mentions the 2nd spin as their second bullet point, and in fact it's pretty simple -- technologies to instant recover backups is pretty common and for virtualization it usually works by mounting the backup files as read-only, then taking a snapshot with a hypervisor, then performing the deletions -- the original data in the backup file is untouched and you can perform the deletions, but the deletions only happen on the snapshot's scope -- so the restored data indeed is free and clear of the deleted data, but the backup file is left alone. Once you're done, just delete the snapshot and unpublish the backup file, and it's like nothing changed. You can in fact do the same with File based backups as usually there is some backup catalog anyways, and you can just flag items as "DNR" and the software obliges during restores. "Not within reach" I'm not sure is specifically clarified by any discussion I've seen, but the common interpretation of GDPR is that more restricted access to any personal data (or copy of the data) is required, and that companies have to publish a bit more about how their processing of collected data works. The industry itself sits pretty comfortably on tons of myths and legends about GDPR and backups, which is sad as GDPR itself really doesn't specifically mention backups, it mentions about personal data that is public and the data collection policies. My hot take is that most companies have purposefully just dragged their feet at best and never stopped doing the stuff that data retention policies were meant to stop, or at worst just decided to annoy users with unnecessary banners and "consent" forms (and indeed, I put full blame for those on the companies -- there's no such requirement to harass and annoy users with a maze of consent options; quite the opposite, they should be able to opt in/out of the consent with ease and without harassment).
xyzzy123 5y ago
Most regulations require "reasonable effort" and a final judgement really only comes from an auditor / discussion with regulators. Honestly even that is often not final except in the context of one particular audit, "reasonable" can differ from auditor to auditor and understandings can change over time. Deletion is hard. For example, deleting a record from a database often leaves data on the underlying storage. Database backups in cloud are often made at the volume level which poses obvious difficulties. On an SSD or log structured fs, even when "overwritten", data may not actually be deleted until the device gets around to re-using a particular physical block. If you are using a service like cloud-managed kafka ALL of these things might be in play at the same time! With the additional constraint that you have even less access to the underlying resources than usual. The safest way to "really delete" say, PII in 2022 is only store it encrypted, burn the key on delete, issue deletes where you need to and hope for the best. Storing sensitive information in a dedicated place and tokenizing it / using links to limit data spreading and copying can also make your "compliance story" a lot simpler. But no regulator I know of will make you do this, reasonable effort is the key. Almost universally "hard" db delete + an explanation of when stuff will age out of backups would be fine. It would not be "reasonable" to demand redaction of system backups.
- dataflow 5y ago
  Random thought: what if the criterion for deletion was the following? "Made sufficiently inaccessible so that, even if validly subpoenaed by a court of competent jurisdiction, you would respond that you have no practical means for recovering the data."
  - csydas 5y ago
    It's not really a clear statement, and I think it also sort of misses the goal of deletion policies. The above statement for me reads with the implication that it only checks accessibility by the party responsible for the data, as the example check is that the responsible party has no practical means to recovery it. But in fact this is not really a good goal I think as it doesn't really protect against what data retention laws aim for; avoid keeping data you don't need and most definitely don't sell it off. Thus, an attacker looking to do a DB dump will most definitely engage in activities that are not practical to recover inaccessible data; a data sales team will certainly modify the data to still be equally identifiable and targeted but transformed enough that the original dataset is no longer required. It really doesn't address what a lot of data deletion/retention policies aim to succeed with.
deleted 5y ago
[deleted]

69 points by nikanj 5y ago

mhitza 5y ago

As long as your end users are well informed about your backup retention policy, and when restoring from those backups you make sure to delete the user data; I see no issue.

paulsutter 5y ago

Back up encrypted data and discard keys selectively to delete data from the archive

bottled_poe 5y ago
This is an excellent solution. I don’t see why it couldn’t be the industry standard.
- Silhouette 5y ago
  It's a solution that looks clever at first sight but the devil is in the details. Sometimes it's difficult to determine what data is subject to erasure in the first place, for example if there are multiple parties involved or the obligation to erase conflicts with other obligations that might require retention of some of the personal data. If you're going with the encryption and key management strategy, your design must accommodate all possible scenarios both now and in the future including those where the rules change between now and then. For anything but the simplest cases that seems unlikely.
  - bottled_poe 5y ago
    Seems easy to me. 30 days, delete backup keys. What’s the problem? You still have 29 day old backups?
    - Silhouette 5y ago
      Maybe we're talking about slightly different things here. The suggestion to use individual keys to encrypt personal data is usually made so that deleting a key has effectively and immediately destroyed all copies of the data including any historical ones. That way you're definitely fully compliant with any right to be forgotten requests with one simple act. Even if you do have some older backups or archives kept around or the backup data isn't truly irretrievable from the physical media where it was stored after being marked for deletion, that data instantly becomes random noise without the key (which you obviously don't store and backup in the same way in this kind of scheme). In other words, it's not really about whether keeping backups for a few days or a few weeks before they are "deleted" would be considered acceptable by regulators, it's about making all such questions irrelevant while otherwise being able to continue with business as usual. It would be a clever and useful solution to the regulatory compliance problem in this area, if it worked in practice.
- jandrewrogers 5y ago
  It doesn't work in real systems because it is extremely costly to the point of being infeasible. This has been known for decades and rejected because it doesn't actually work. If a technical solution is "obvious" and no one is using it, it tends to imply that it has serious flaws that are obvious to domain experts. Using encryption for deletion is known to be deeply flawed for unavoidable technical reasons. These databases actually exist. Almost no one uses them because the performance is exceptionally poor to the point of being nearly unusable for practical things.
  - bottled_poe 5y ago
    I disagree. Why doesn’t time-bounded deletion of backup keys work?
dogma1138 5y ago
That doesn’t scale, if you need to delete records pertaining to a specific person and you have a user base of 10M yet alone 100M or 2B users you can’t encrypt each of their records with an individual key.
- bottled_poe 5y ago
  I think the parent is suggesting discarding keys for all backups after X days. So, anything deleted in the live environment effectively gets removed from the backups after X days.
- lucb1e 5y ago
  Depends on how you back up. If you copy the raw database files that you also run queries on (e.g. take a zfs snapshot), then you'd need some very special encryption to be able to query across differently-encrypted data. But if your database is just on an encrypted disk but otherwise readable, and you only encrypt the records on an individual level when exporting them to the backup system... that can work. Generating a secure random encryption key after seeding can be as fast as a single AES function call, and symmetric encryption isn't particularly expensive either. Still extra work compared to just copying bytes between memory regions (now everything has to be operated on by the CPU), I'm not saying it's going to be trivial, but it doesn't seem infeasible. Also at 2B users as you mention, which is an extreme example considering we have only a few billion internet users (multiply by one or two bots for every person, but even then), you need huge storage and compute clusters anyway. Storing 16 bytes of key material extra isn't going to be the difference between feasible and infeasible. It doesn't really matter if you're talking 2e2 or 2e9 users in terms of computational or storage overhead.
- benlivengood 5y ago
  Large companies actually do keep individual master keys per user. Think of it this way, what's an extra 256 bits (32 bytes) stored for every user in a system when their name and email address probably exceeds 32 bytes already? Master keys encrypt the records of a user-key database where user keys are stored. User keys protect unique wrapped keys that are randomly generated at time of encryption (perhaps this is where your logistical argument arose) for individual pieces of data. This is commonly implemented as an encryption service that will encrypt/decrypt blobs of data for services on behalf of a particular user via RPC; the cryptographic keys are never present in normal service code or standard servers and only within the encryption service (which may be an HSA) so there's a much smaller domain to secure. Common ACLs protect calls to the encryption service per user, like they protect access to storage and other resources. Obviously for bulk data encryption this is less tenable so services handling bulk data (many GB) get extra protection and direct access to unwrapped user keys, but must be responsible for respecting privacy/security ACLs internally.
contravariant 5y ago
This might be appropriate for some cases, but is dangerously similar to just not making backups in the first place.
SnowHill9902 5y ago
If I use a 16 bit key does it count as deleted data?
- lucb1e 5y ago
  Roughly equivalent to hashing a small-ish number like a phone numbers, so based on [1] I'd say yes! Marketing can now say with a straight face that through this technical operation, basically sprinkling some magic pixie dust over your data, it is now incomprehensible and no longer leads back to you. [1] 7:05 - 7:58 https://projectbullrun.org/surveillance/2015/video-2015.html#bernstein https://projectbullrun.org/surveillance/2015/video-2015.html... (2015)
twhitmore 5y ago
It just moves the problem, to how do you delete the keys & manage backups of the keys. You have to backup the keys, since a system outage would otherwise leave you unable to function. My shot from the hip: encryption/keying is irrelevant except for highly sensitive data (credit cards, crypto wallets). Plain deletion from production databases should be sufficient, backups should be "at arms length" but otherwise should not need to be treated specially. Another shot from the hip: System outage & RPO loss of a requested deletion is an act of God and no worse (indeed probably less important) than losing the other business transactions in that time period. I don't see special handling being needed for deletion.

lucb1e 5y ago

This seems like a false dichotomy or, at best, a very vague discussion if you don't point out a specific regulation to run afoul of. Taking GDPR and its ~1995 predecessor as an example, which both required personal data to be collected only when there was a reason for it, thus when you have no more use for it, it has to be deleted. However, backups are a perfectly legit reason. It's not an excuse to keep data into eternity, but checking that all systems are good after a delete run, or having a buffer period of some defined time, is definitely reasonable and should (not a lawyer) not run afoul of the regulation. So I am not sure what exactly it is you are concerned about.

Terry_Roll 5y ago

>How do various regulators view this? Its a good question, one I've mentioned on YV in the past. I havent seen it handled in the UK, but the backup industry isnt handling it either from what I've seen. I think the regulators will view it as it needs to deleted from backups, but the backup software industry doesnt really make it easy to delete stuff from said backups. It could be done because individual file & mailbox restores exist so if they can identify what needs to be restored, they could just as easily delete it.

macintux 5y ago

Title should be prefaced with “Ask HN:”

colonelxc 5y ago

It's generally not required to 'permanently delete instantly' (this includes GDPR). So you take a backup every day, and delete backups after 30 days (or whatever amount of time you need to for compliance). When somebody says 'delete this permanently', you do, from your primary data store. Then after 30 days, not even the backups will have anything. You can also get fancy with more than one stage (soft delete, hard delete, then backup purge), as long as the whole pipeline is done within the compliance window. Of course, the trickiest part of all of this is knowing where all data can proliferate, and making sure it doesn't (don't let people download data to their machines, don't let people create a 'copy of prod' that sticks around outside or retention.

phire 5y ago

I'm of the opinion that as long as a company has policies to never restore legitimately deleted data from backups, then backups of deleted data simply shouldn't count. Everything else is just too ugly to implement. Though I get the impression that regulators are not happy with this opinion, and what that data to be physically deleted within some timeline.

Silhouette 5y ago

In the UK, the regulator has a specific comment on this exact issue: https://ico.org.uk/for-organisations/guide-to-data-protection/guide-to-the-general-data-protection-regulation-gdpr/individual-rights/right-to-erasure/#ib5 https://ico.org.uk/for-organisations/guide-to-data-protectio...

netman21 5y ago

A technical reading of GDPR leads one to think that data in a backup should also be deleted. But in practice most orgs are treating Article 7, The Right to Erasure, as meaning "make no longer publicly available." That still leaves the issue of ensuring processes are in place to make sure the "deleted" data is not restored from backups.

notreallyserio 5y ago

Oof, only now do I realize I need to record account deletions somewhere independent of my database backups so I don't accidentally restore accounts I shouldn't if something goes horribly wrong. Not sure why I didn't think of this before.

kingcharles 5y ago
Ah, this does add a layer of complexity I hadn't thought of before.

Nextgrid 5y ago

Legitimate interest (as mentioned in the GDPR) is your friend here. GDPR enforcement is not black and white and there is no specific rules or timeframes you’re allowed to keep data - instead, if there is a complaint, the regulator will work with you to understand your argument before any penalties are handed out, so you’ll be fine unless you’re obviously acting in bad faith. So deleting data from the primary data store while keeping backups for a reasonable amount of time (3 months?) is totally fine even though you are technically right in that the data hasn’t yet been fully deleted.

tedk-42 5y ago

A 'Mark for deletion' coupled with a check once the deletion marker should be processed has succeeded would be a way to avert what happened at Atlassian. No matter how well built a system, human error can still happen which leads us down dark paths. Also, no backup currently exists for when an alien race blows up the Earth. Some things are simply impossible to account for.

zinodaur 5y ago

A pattern that has saved my bacon a number of times: When writing the "permanent delete" code, only issue the delete when you have proven that there is a backup. i.e.: - Create a backup of the data you want to delete and store it somewhere - Read everything in the backup, and delete everything you find in the backup from prod - Have the backup expire after a certain time threshold (e.g. s3 has a nice TTL setting for backup buckets) Even if you delete the wrong thing, the backups are always immediately accessible in the near-term

ledgerdev 5y ago
I like this, excellent way to handle permanent record deletion.

tgsovlerkhgsel 5y ago

First of all, most privacy regulations are surprisingly toothless and rarely enforced, so in practice, you get away with anything until you somehow do draw the ire of a regulator, and then they'll enforce their random favorite pet rules that don't make much sense, and may also change arbitrarily. (Think "must use fax, can't use e-mail unless it's PGP-encrypted" followed by "can't use fax either".) German regulators that I spoke with acknowledge the need for backup, but require you to keep a separate "deletion log", so that when data is restored from backup, you can replay the deletions and keep supposed-to-be-deleted data from entering live systems. You also can't keep the backups forever, but if you have a reasonable plan, you're most likely going to be OK (even if they disagree, they'll probably just tell you to change your retention periods). This applies to privacy in general - as long as you show a minimum of genuine care, and deal with a few formal requirements, you're probably going to be OK. GDPR is most scary for people who a) think it will be enforced to the fullest extent that a pessimistic reading of the law might allow, or b) are actively doing privacy-hostile things that they know they shouldn't be doing but want to keep doing. For example, storing the data for backup purposes, then going back in and using it to build ML models because the backups have more data than the live system would (rightfully) have a decent chance of getting you in trouble. You can keep data for backup purposes, but if you're allowed to store data for purpose A, that doesn't mean "I'm allowed to store it so now I can use it for any purpose" as some people seem to think.

ledgerdev 5y ago

> What are the industry best practices for data deletion? Tokenization. Exchange the data for tokens, and store the sensitive data someplace very secure separate from the main operational system. To process a delete, delete the token. Ideally this is handled via proxy, so it's a completely separate system from primary app. Having used this method, I feel it ought to be used for every app storing sensitive data. edit: I forgot to mention the biggest issue I have with crypto shredding (encryption per record) is that you can't correlate data between records as you can with tokens.

kevin_nisbet 5y ago
It's not clear to me that this is really sufficient, or even really answers the original question. On the sufficient side, I don't know that retaining an encrypted version of customer data would really meet a lot of compliance definitions for deleted data. Atleast, nothing I've heard of tested in court. Also, depending on the data, even tokenized / encrypted / etc approaches are easy to get wrong and still allow some correlation. In a previous company when it came to network tooling, we had to be extremely careful that we didn't intercept 1 bit of payload, as even though that 1 bit couldn't be used to meaningfully reproduce a phone call, by many legal definitions we would be illegally intercepting phone calls. And then circling back to the original question, the problem of how to manage data and customer deletions if just shifted to how to do this with the tokens. So my take on the original question is as follows. To my knowledge, the right to be forgotten compliance regimes do include time windows for deletions, understanding that it can take time to purge requested data. Also, some customers may want this in their contracts, so I'd try and align any contracts / terms of service with the most strict compliance regimes. When taking backups, if possible use WORM (on S3 I think this is called object locks), and set the lock to be less time than the compliance regime requires, but long enough that there is always locked backups that can't be deleted by a mistake or adversary. Depending on required history, you can then just let the backups expire. If you need longer backup history than the compliance regimes allow, then it comes to structuring backups so selective data can be removed, and then relocking the backup. And doing this on a rolling basis, so at no point in time or run of the cleanup are all backups being touched at the same time.
TazeTSchnitzel 5y ago

dasil003 5y ago

You don’t. Restoring a backup is rare. If this happens to restore a deleted data, the deletion request is part of the lost data. Trying to work around this with a separate delete request log doesn’t solve the problem, it just shrinks the loophole. But if you are restoring often enough for this to draw the ire of regulators you probably have much bigger problems. Of course I’m just giving you the technical truth straight up. The other part of this is you would never say it so directly to a regulator or auditor. Work with your compliance team to get the messaging right.

zeckalpha 5y ago
Unless they are different databases.

swiftcoder 5y ago

The way most bigger companies deal with this is by making all deletions soft-deletes (i.e. just set a deleted=true flag in the database record), and then scheduling the hard deletion as a cancellable async process to be run some time in the future (say, 90 days). It's very hard to cause massive outages with soft deletes, it's easy to rollback, and compliance rules rarely put a tight deadline on mandatory deletions (i.e. that 90 day deletion window is GDPR compliant)

jasonkester 5y ago

My approach involves two steps: 1. Never permanently delete anything. 2. Never implement anything that makes your product worse because of some silly regulation. So if you “delete” your cat picture from one of my sites, I’ll flip the IsActive bit to false for you and nobody will ever see it again. Then when you realize that it was your favorite cat picture and how could you delete it forever I never told you to do that I’ll sue you, I can flip that IsActive bit back to true. That situation has happened way more often than the one where the New Zealand Data Storage Task Force Enforcement Division has levied fines against me. So it’s the best strategy.

furstenheim 5y ago
GDPR fines are pretty hefty. I doubt the employer would be happy with soft deletion. Also, you don't need an auditor. In the eventuality of a hack, the db could be made open and then it would become clear you were not deleting
TazeTSchnitzel 5y ago
This is a very bad idea for GDPR compliance, but it is an even worse idea if you don't want to be charged with possession of CSAM.

traceroute66 5y ago

Forgive me, but I'm not sure why this is even a question. Compliance comes first, because if you don't comply with legislation you most likely won't be in business for long. So you then build on from that by designing your system to take compliance aspects into account. If you do that, then the backup question no longer occurs, because your system design has taken into account the fact that (for example) GDPR "Right to be forgotten" exists, and so you will have ensured you have a mechanism in place to ensure the "forgotten" PII does not make its way into your backups.

ealexhudson 5y ago

Backing up data that is later permenantly deleted is fine. From a regulatory point of view, typically you only need to ensure: * the backups are not "within reach" and that there's no danger of data that has been deleted being looked up/used. * any restoration of backups will not result in deleted data coming back to life. This is the tougher requirement, and means you do need to be able to re-delete things somehow - but typically this is only an issue in scenarios where you're having to undo damage that went undetected for a long time, so you have other big fish to fry also. There are often competing compliance concerns here. E.g. a legal hold on data should prevent stuff being deleted, and in some instances may allow you to go back into backups to retrieve deleted things.

csydas 5y ago
I want to second and elaborate on the above statement. As I work heavily with backups, this is something that is fraught with a lot of arm-chair analyses, which I will try to avoid, and instead focus on just how the industry has reacted. For Backups, most vendors have two takes on this, and my hot take is that it's less about legal interpretations and it's how the vendors spin their backup technology features: 1. Vendors that have mutable backups (e.g., the backup archive content can be changed and individual records, file, and other stuff can be edited/removed) finally have a defense against the security risk of mutable backups and now it's a compliance feature 2. Vendors with non-mutable backups (e.g., the backup file itself is still mutable (e.g., no immutable bit), but you cannot remove individual records/entries from it without destroying the backup file itself) have simply developed staging restores that allow for programatic removal of a non-mutable restored machine before performing an actual restore. The parent mentions the 2nd spin as their second bullet point, and in fact it's pretty simple -- technologies to instant recover backups is pretty common and for virtualization it usually works by mounting the backup files as read-only, then taking a snapshot with a hypervisor, then performing the deletions -- the original data in the backup file is untouched and you can perform the deletions, but the deletions only happen on the snapshot's scope -- so the restored data indeed is free and clear of the deleted data, but the backup file is left alone. Once you're done, just delete the snapshot and unpublish the backup file, and it's like nothing changed. You can in fact do the same with File based backups as usually there is some backup catalog anyways, and you can just flag items as "DNR" and the software obliges during restores. "Not within reach" I'm not sure is specifically clarified by any discussion I've seen, but the common interpretation of GDPR is that more restricted access to any personal data (or copy of the data) is required, and that companies have to publish a bit more about how their processing of collected data works. The industry itself sits pretty comfortably on tons of myths and legends about GDPR and backups, which is sad as GDPR itself really doesn't specifically mention backups, it mentions about personal data that is public and the data collection policies. My hot take is that most companies have purposefully just dragged their feet at best and never stopped doing the stuff that data retention policies were meant to stop, or at worst just decided to annoy users with unnecessary banners and "consent" forms (and indeed, I put full blame for those on the companies -- there's no such requirement to harass and annoy users with a maze of consent options; quite the opposite, they should be able to opt in/out of the consent with ease and without harassment).

xyzzy123 5y ago

Most regulations require "reasonable effort" and a final judgement really only comes from an auditor / discussion with regulators. Honestly even that is often not final except in the context of one particular audit, "reasonable" can differ from auditor to auditor and understandings can change over time. Deletion is hard. For example, deleting a record from a database often leaves data on the underlying storage. Database backups in cloud are often made at the volume level which poses obvious difficulties. On an SSD or log structured fs, even when "overwritten", data may not actually be deleted until the device gets around to re-using a particular physical block. If you are using a service like cloud-managed kafka ALL of these things might be in play at the same time! With the additional constraint that you have even less access to the underlying resources than usual. The safest way to "really delete" say, PII in 2022 is only store it encrypted, burn the key on delete, issue deletes where you need to and hope for the best. Storing sensitive information in a dedicated place and tokenizing it / using links to limit data spreading and copying can also make your "compliance story" a lot simpler. But no regulator I know of will make you do this, reasonable effort is the key. Almost universally "hard" db delete + an explanation of when stuff will age out of backups would be fine. It would not be "reasonable" to demand redaction of system backups.

dataflow 5y ago
Random thought: what if the criterion for deletion was the following? "Made sufficiently inaccessible so that, even if validly subpoenaed by a court of competent jurisdiction, you would respond that you have no practical means for recovering the data."
- csydas 5y ago
  It's not really a clear statement, and I think it also sort of misses the goal of deletion policies. The above statement for me reads with the implication that it only checks accessibility by the party responsible for the data, as the example check is that the responsible party has no practical means to recovery it. But in fact this is not really a good goal I think as it doesn't really protect against what data retention laws aim for; avoid keeping data you don't need and most definitely don't sell it off. Thus, an attacker looking to do a DB dump will most definitely engage in activities that are not practical to recover inaccessible data; a data sales team will certainly modify the data to still be equally identifiable and targeted but transformed enough that the original dataset is no longer required. It really doesn't address what a lot of data deletion/retention policies aim to succeed with.

deleted 5y ago

[deleted]