3 ms·

I'd be surprised to hear about anybody deleting records on backup systems (i.e., backups that exist for disaster recovery purposes). It would be pretty difficul

I'd be surprised to hear about anybody deleting records on backup systems (i.e., backups that exist for disaster recovery purposes). It would be pretty difficult to do that in a commercially viable way (but I'm open to hearing about some creative ideas).

bombcar 3y ago
There are methods of doing it, but it is complex. Basically you either have hot backups you can delete from (this is bad for obvious reasons), or your backups expire in a given time (this is most common), or you have each record encrypted with an encryption key that is saved in other ways so you only have to destroy those to make the data irretrievable. Of course, that system has to be backed up, etc, etc.
plagiarist 3y ago
If user data in backups is encrypted with a key kept in warm/hot storage, you can delete the key and effectively delete data in cold storage because it is unrecoverable. This obviously must be per-user keys and set up in advance so I doubt they have it available, but that's one potential method.
- starttoaster 3y ago
  Where do the keys get backed up to? If nowhere: If the key system fails, do you just accept the loss of _all_ of your customer data? If somewhere: by what mechanism do you delete the key in a backup?
  - caconym_ 3y ago
    The keys dataset is going to be much smaller than everything else, and possibly immutable per customer. This in principle makes it much simpler and cheaper to handle, so it's easy to imagine how---depending on the scales and technologies involved---one could isolate it and achieve the desired redundancies and retention periods in ways that would be impractical with the full customer dataset.
    - starttoaster 3y ago
      That’s a long way to say “it’s definitely possible but I haven’t architected a sane solution yet.”
      - caconym_ 3y ago
        I find this comment rather baffling. Can you really not see what I'm talking about without having a concrete (and likely completely irrelevant to 23andMe's requirements) example spelled out for you in detail?
        starttoaster 3y ago
        You didn’t even give vague details, let alone you giving a concrete example in detail. I asked a question and your previous comment answer boils down to basically, “it’s totally possible.” My theory is that you’re making a concession somewhere in the backups to a separate keyring system. Either there is no cold backup, or you don’t do cold backups at all, or your cold backup is actually semi-warm and needs to be hooked up to a system intermittently to be reconciled against production (in which case, the backups need backups to protect against a failure on the reconciler system.) The onus is on the answerer to tell me how they would avoid one of those concessions. Respectfully, anything else/less is just fluff like, “it’s totally possible.”
        caconym_ 3y ago
        The whole point of isolating the keys dataset is that you can reason about it differently. You frame these "concessions" as dealbreakers, but I don't think that's supported. Can you explain why this dataset would need cold snapshots retained past a period users would accept as a delay for guaranteed account deletion---say, two weeks? Replication already has you covered on acts of god, so we're worried about things like bad code pushes, "hackers", and so on.
        starttoaster 3y ago
        > so we're worried about things like bad code pushes, "hackers", and so on. I'm confused. Are you saying those are small concerns? Because I'm saying the backup mechanism for the keys surely need to be resilient to all of those.
pbhjpbhj 3y ago
Isn't it required to be available by EU law?
- gwright 3y ago
  Apparently GDPR doesn't cover this case specifically but several enforcement authorities have issued some guidance. The only reasonable approach that I've seen is to maintain a log of deletion requests and ensure that if a backup is used to restore operational data that the deletion request is applied against the restored system. Ironically, I've responded to deletion requests made by email in which the person did not have any records in our systems, until receiving the deletion request containing their name and email address. https://verasafe.com/blog/do-i-need-to-erase-personal-data-from-backup-systems-under-the-gdpr/ https://verasafe.com/blog/do-i-need-to-erase-personal-data-f...

bombcar 3y ago

There are methods of doing it, but it is complex. Basically you either have hot backups you can delete from (this is bad for obvious reasons), or your backups expire in a given time (this is most common), or you have each record encrypted with an encryption key that is saved in other ways so you only have to destroy those to make the data irretrievable. Of course, that system has to be backed up, etc, etc.

plagiarist 3y ago

If user data in backups is encrypted with a key kept in warm/hot storage, you can delete the key and effectively delete data in cold storage because it is unrecoverable. This obviously must be per-user keys and set up in advance so I doubt they have it available, but that's one potential method.

starttoaster 3y ago
Where do the keys get backed up to? If nowhere: If the key system fails, do you just accept the loss of _all_ of your customer data? If somewhere: by what mechanism do you delete the key in a backup?
- caconym_ 3y ago
  The keys dataset is going to be much smaller than everything else, and possibly immutable per customer. This in principle makes it much simpler and cheaper to handle, so it's easy to imagine how---depending on the scales and technologies involved---one could isolate it and achieve the desired redundancies and retention periods in ways that would be impractical with the full customer dataset.
  - starttoaster 3y ago
    That’s a long way to say “it’s definitely possible but I haven’t architected a sane solution yet.”
    - caconym_ 3y ago
      I find this comment rather baffling. Can you really not see what I'm talking about without having a concrete (and likely completely irrelevant to 23andMe's requirements) example spelled out for you in detail?
      - starttoaster 3y ago
        You didn’t even give vague details, let alone you giving a concrete example in detail. I asked a question and your previous comment answer boils down to basically, “it’s totally possible.” My theory is that you’re making a concession somewhere in the backups to a separate keyring system. Either there is no cold backup, or you don’t do cold backups at all, or your cold backup is actually semi-warm and needs to be hooked up to a system intermittently to be reconciled against production (in which case, the backups need backups to protect against a failure on the reconciler system.) The onus is on the answerer to tell me how they would avoid one of those concessions. Respectfully, anything else/less is just fluff like, “it’s totally possible.”
        caconym_ 3y ago
        The whole point of isolating the keys dataset is that you can reason about it differently. You frame these "concessions" as dealbreakers, but I don't think that's supported. Can you explain why this dataset would need cold snapshots retained past a period users would accept as a delay for guaranteed account deletion---say, two weeks? Replication already has you covered on acts of god, so we're worried about things like bad code pushes, "hackers", and so on.
        starttoaster 3y ago
        > so we're worried about things like bad code pushes, "hackers", and so on. I'm confused. Are you saying those are small concerns? Because I'm saying the backup mechanism for the keys surely need to be resilient to all of those.

pbhjpbhj 3y ago

Isn't it required to be available by EU law?

gwright 3y ago
Apparently GDPR doesn't cover this case specifically but several enforcement authorities have issued some guidance. The only reasonable approach that I've seen is to maintain a log of deletion requests and ensure that if a backup is used to restore operational data that the deletion request is applied against the restored system. Ironically, I've responded to deletion requests made by email in which the person did not have any records in our systems, until receiving the deletion request containing their name and email address. https://verasafe.com/blog/do-i-need-to-erase-personal-data-from-backup-systems-under-the-gdpr/ https://verasafe.com/blog/do-i-need-to-erase-personal-data-f...