4 ms·

You can sleep soundly: it seems like they back _everything_ up: > Second, the script we used provided both the "mark for deletion" capability ... (where reco

You can sleep soundly: it seems like they back _everything_ up: > Second, the script we used provided both the "mark for deletion" capability ... (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons. The script was executed with the wrong execution mode and the wrong list of IDs. The result was that sites for approximately 400 customers were improperly deleted. > To recover from this incident, our global engineering team has implemented a > methodical process for restoring our impacted customers. [https://www.atlassian.com/engineering/april-2022-outage-update https://www.atlassian.com/engineering/april-2022-outage-upda...] Anyone else find it disturbing that they are able to restore data that they deleted permanently for "compliance" reasons? If this is true, how were they ever compliant? I guess data is only permanently deleted when the engineering team is following their typical, non-methodical process...

notatoad 5y ago
No, I don't think that's disturbing. That's the point of backups - even when something is permanently and completely erased in the production database, it's still in the backup. Eventually it will get rotated out as the backups expire. Going back and purging things from the backups as part of the delete process would be overdoing it to a ridiculous degree.
- yebyen 5y ago
  I think that depends on what you mean by compliance. Some regulations require you to irreversibly destroy data when they prescribe the destruction of that data. That can mean as much as "you have to encrypt everything with a separate key, so that you can destroy the key for the given (say, personally identifiable) dataset making its retrieval irrecoverable" I'm not saying that's the particular compliance reason they had here, or that the analysis you're giving is wrong, either. There is an interpretation where either of these ideas could be the correct one.
  - nindalf 5y ago
    This is why regulations specify that data must be destroyed within a time period, typically 90 days. It gives enough time for backups to rotate out. If this weren’t a concern, regulations would demand immediate deletion of data.
  - a2800276 5y ago
    "permanently delete" strongly suggests to me that it was the "medical and financial data" kind of compliance. If data can be restored, it's not permanently deleted. But this was a statement from the CEO, so words can have arbitrary meaning :)
    - notatoad 5y ago
      "permanently delete" does not mean the same thing as "immediately delete". deleting from the live database is the first step of a permanent deletion, as long as the data exists somewhere the deletion process is still in-progress. there's a whole lot of people in here who are way too quick to assume that just because one part of a permanent deletion process was inadvertently triggered and then caught while they still had backups, their whole permanent deletion process is a lie.
      - voxic11 5y ago
        https://ico.org.uk/for-organisations/guide-to-data-protection/guide-to-the-general-data-protection-regulation-gdpr/individual-rights/right-to-erasure/#ib5 https://ico.org.uk/for-organisations/guide-to-data-protectio... You seem to be right-ish, while the gdpr in certain circumstances allows you to keep backups of data that should have been deleted it seems like they are trying to discourage it in the future. > ...It is, however, important to note that where data put beyond use is still held it might need to be provided in response to a court order. Therefore data controllers should work towards technical solutions to prevent deletion problems recurring in the future.
        deleted 5y ago
        [deleted]
- Delitio 5y ago
  Nope it's not ridiculous. If you are only allowed to store data for x month that's it. It's your job to use technics which allow you to do this like using encryption on your backup and deleting the keys for it, for example.
  - scottlamb 5y ago
    Delitio> If you are only allowed to store data for x month that's it. Exactly. I'm not aware of any laws saying "you must delete this data immediately". More like "within X days or months". The permanently delete thing presumably skips some cooling-off period in the online database but not the backup, which seems perfectly appropriate, provided your backup retention is compliant. Google has a nice page describing out their deletion process. [1] It doesn't go into product-specific technical details/steps (like marked as deleted within the product, row deleted from Bigtable/Spanner, major compaction guaranteed to happen, backups guaranteed to be deleted or unusable) but it says this: Google> We then begin a process designed to safely and completely delete the data from our storage systems. Safe deletion is important to protect our users and customers from accidental data loss. Complete deletion of data from our servers is equally important for users’ peace of mind. This process generally takes around 2 months from the time of deletion. This often includes up to a month-long recovery period in case the data was removed unintentionally. This is a best practice. Delitio> It's your job to use technics which allow you to do this like using encryption on your backup and deleting the keys for it, for example. If they'd thrown away the encryption key immediately, this would have been much worse. Instead of "we're down for 2 weeks?!?" (already quite bad) it'd be "our data is gone forever?!?". You never want to delete anything too quickly for exactly this reason. [1] https://policies.google.com/technologies/retention?hl=en-US https://policies.google.com/technologies/retention?hl=en-US
- usefulcat 5y ago
  > Going back and purging things from the backups as part of the delete process would be overdoing it to a ridiculous degree. Also, modifying backups is a great way to inadvertently hose your backups.
luhn 5y ago
It's generally recognized that deleting data from a backup would violate the integrity of the backup, so allowances are made. Usually you have to make sure the data is deleted as part of the restore process. For example, from CCPA: > If a business stores any personal information on archived or backup systems, it may delay compliance with the consumer's request to delete, with respect to data stored on the archived or backup system, until the archived or backup system relating to that data is restored to an active system or next accessed or used for a sale, disclosure, or commercial purpose.
deepspace 5y ago
I asked the same question yesterday, and the responses were food for thought. If you make backups, you are, almost by definition, unable to perform a full 'Compliance Delete' before the oldest backup in the set has expired. Compliance-based deletion, if it is offered as a service, is almost always something time-based, like "we guarantee the data will be deleted 7 years from now". And then that deliberate deletion step is baked into the backup process. So, i.m.o. at best they misrepresented the nature of the compliance deletion process. It never did what it was designed to do.
twistedpair 5y ago
One assumes the backups will rotate out and be discarded after a fixed time, so everything is "truly deleted" for compliance reasons.
dub 5y ago
Generally user data deletion happens in multiple phases for large companies that care about both compliance and user experience. For example, if you delete an email or document on Google it moves to the "Trash" folder for 30 days. When you manually empty the trash or the time window expires, most likely the next step would be a soft deletion for a few days where the data is still on hard drives but hidden from the application. Soft deletion is mainly protection against coding errors, since soft deletion is easy to undo if you've caused an incident but hard deletion (removing the data from disk) is not. Then most likely a garbage collection process comes by a few days later and hard deletes the data from disk, leaving it only on tape backups Finally, maybe a month or two later it disappears from the tape backups as they get rotated or otherwise disposed of This addresses the needs of: - Giving a good user experience (user "oops I made a mistake" undelete) - Protecting against incidents due to coding errors (software engineer "oops I made a mistake" undelete) - Making sure data disappears from both disk and backups within a certain time window, like maybe 30 or 60 days (comply with regulation and user expectations of data being cleared)
- dub 5y ago
  Another option for clearing tape backups is to throw away the encryption keys, as https://www.youtube.com/watch?v=ejBncCrlAqc https://www.youtube.com/watch?v=ejBncCrlAqc mentions
donavanm 5y ago
> Anyone else find it disturbing that they are able to restore data that they deleted permanently for "compliance" reasons? An overarching theme with these things is “legitimate business need” and “no indefinitely retained customer data. Having backups, system event logs, etc are all legitimate business needs. Based on the data type that business need may be days or years with things like financial and legal requirements. Youre conflating permanent, immediate, and irrevocable. These are usually handled in different aspects. Think of accounts having multiple states like active, suspended, closed, terminated, purged. Some examples; suspended: credentials/authnz immediately disabled, all data online, charges continue to accrue, can be restored in minutes. Closed: credentials disabled, data online, processing stopped, charges stopped, may take manual intervention (hours) to return to active. Terminated: creds & account irrevocably unavailable, online data deleted, offline data (backups) remains available. Purged: all online and offline customer data irrevocably unavailable. This generally happens after a defined retention period for things like logs, backups, etc. You can apply similar concepts to individual resources more granularly than the account. Disclaimer: principal at AWS but the above is my own opinion/observation and does not represent my employer.

notatoad 5y ago

No, I don't think that's disturbing. That's the point of backups - even when something is permanently and completely erased in the production database, it's still in the backup. Eventually it will get rotated out as the backups expire. Going back and purging things from the backups as part of the delete process would be overdoing it to a ridiculous degree.

yebyen 5y ago
I think that depends on what you mean by compliance. Some regulations require you to irreversibly destroy data when they prescribe the destruction of that data. That can mean as much as "you have to encrypt everything with a separate key, so that you can destroy the key for the given (say, personally identifiable) dataset making its retrieval irrecoverable" I'm not saying that's the particular compliance reason they had here, or that the analysis you're giving is wrong, either. There is an interpretation where either of these ideas could be the correct one.
- nindalf 5y ago
  This is why regulations specify that data must be destroyed within a time period, typically 90 days. It gives enough time for backups to rotate out. If this weren’t a concern, regulations would demand immediate deletion of data.
- a2800276 5y ago
  "permanently delete" strongly suggests to me that it was the "medical and financial data" kind of compliance. If data can be restored, it's not permanently deleted. But this was a statement from the CEO, so words can have arbitrary meaning :)
  - notatoad 5y ago
    "permanently delete" does not mean the same thing as "immediately delete". deleting from the live database is the first step of a permanent deletion, as long as the data exists somewhere the deletion process is still in-progress. there's a whole lot of people in here who are way too quick to assume that just because one part of a permanent deletion process was inadvertently triggered and then caught while they still had backups, their whole permanent deletion process is a lie.
    - voxic11 5y ago
      https://ico.org.uk/for-organisations/guide-to-data-protection/guide-to-the-general-data-protection-regulation-gdpr/individual-rights/right-to-erasure/#ib5 https://ico.org.uk/for-organisations/guide-to-data-protectio... You seem to be right-ish, while the gdpr in certain circumstances allows you to keep backups of data that should have been deleted it seems like they are trying to discourage it in the future. > ...It is, however, important to note that where data put beyond use is still held it might need to be provided in response to a court order. Therefore data controllers should work towards technical solutions to prevent deletion problems recurring in the future.
      - deleted 5y ago
        [deleted]
Delitio 5y ago
Nope it's not ridiculous. If you are only allowed to store data for x month that's it. It's your job to use technics which allow you to do this like using encryption on your backup and deleting the keys for it, for example.
- scottlamb 5y ago
  Delitio> If you are only allowed to store data for x month that's it. Exactly. I'm not aware of any laws saying "you must delete this data immediately". More like "within X days or months". The permanently delete thing presumably skips some cooling-off period in the online database but not the backup, which seems perfectly appropriate, provided your backup retention is compliant. Google has a nice page describing out their deletion process. [1] It doesn't go into product-specific technical details/steps (like marked as deleted within the product, row deleted from Bigtable/Spanner, major compaction guaranteed to happen, backups guaranteed to be deleted or unusable) but it says this: Google> We then begin a process designed to safely and completely delete the data from our storage systems. Safe deletion is important to protect our users and customers from accidental data loss. Complete deletion of data from our servers is equally important for users’ peace of mind. This process generally takes around 2 months from the time of deletion. This often includes up to a month-long recovery period in case the data was removed unintentionally. This is a best practice. Delitio> It's your job to use technics which allow you to do this like using encryption on your backup and deleting the keys for it, for example. If they'd thrown away the encryption key immediately, this would have been much worse. Instead of "we're down for 2 weeks?!?" (already quite bad) it'd be "our data is gone forever?!?". You never want to delete anything too quickly for exactly this reason. [1] https://policies.google.com/technologies/retention?hl=en-US https://policies.google.com/technologies/retention?hl=en-US
usefulcat 5y ago
> Going back and purging things from the backups as part of the delete process would be overdoing it to a ridiculous degree. Also, modifying backups is a great way to inadvertently hose your backups.

luhn 5y ago

It's generally recognized that deleting data from a backup would violate the integrity of the backup, so allowances are made. Usually you have to make sure the data is deleted as part of the restore process. For example, from CCPA: > If a business stores any personal information on archived or backup systems, it may delay compliance with the consumer's request to delete, with respect to data stored on the archived or backup system, until the archived or backup system relating to that data is restored to an active system or next accessed or used for a sale, disclosure, or commercial purpose.

deepspace 5y ago

I asked the same question yesterday, and the responses were food for thought. If you make backups, you are, almost by definition, unable to perform a full 'Compliance Delete' before the oldest backup in the set has expired. Compliance-based deletion, if it is offered as a service, is almost always something time-based, like "we guarantee the data will be deleted 7 years from now". And then that deliberate deletion step is baked into the backup process. So, i.m.o. at best they misrepresented the nature of the compliance deletion process. It never did what it was designed to do.

twistedpair 5y ago

One assumes the backups will rotate out and be discarded after a fixed time, so everything is "truly deleted" for compliance reasons.

dub 5y ago

Generally user data deletion happens in multiple phases for large companies that care about both compliance and user experience. For example, if you delete an email or document on Google it moves to the "Trash" folder for 30 days. When you manually empty the trash or the time window expires, most likely the next step would be a soft deletion for a few days where the data is still on hard drives but hidden from the application. Soft deletion is mainly protection against coding errors, since soft deletion is easy to undo if you've caused an incident but hard deletion (removing the data from disk) is not. Then most likely a garbage collection process comes by a few days later and hard deletes the data from disk, leaving it only on tape backups Finally, maybe a month or two later it disappears from the tape backups as they get rotated or otherwise disposed of This addresses the needs of: - Giving a good user experience (user "oops I made a mistake" undelete) - Protecting against incidents due to coding errors (software engineer "oops I made a mistake" undelete) - Making sure data disappears from both disk and backups within a certain time window, like maybe 30 or 60 days (comply with regulation and user expectations of data being cleared)

dub 5y ago
Another option for clearing tape backups is to throw away the encryption keys, as https://www.youtube.com/watch?v=ejBncCrlAqc https://www.youtube.com/watch?v=ejBncCrlAqc mentions

donavanm 5y ago

> Anyone else find it disturbing that they are able to restore data that they deleted permanently for "compliance" reasons? An overarching theme with these things is “legitimate business need” and “no indefinitely retained customer data. Having backups, system event logs, etc are all legitimate business needs. Based on the data type that business need may be days or years with things like financial and legal requirements. Youre conflating permanent, immediate, and irrevocable. These are usually handled in different aspects. Think of accounts having multiple states like active, suspended, closed, terminated, purged. Some examples; suspended: credentials/authnz immediately disabled, all data online, charges continue to accrue, can be restored in minutes. Closed: credentials disabled, data online, processing stopped, charges stopped, may take manual intervention (hours) to return to active. Terminated: creds & account irrevocably unavailable, online data deleted, offline data (backups) remains available. Purged: all online and offline customer data irrevocably unavailable. This generally happens after a defined retention period for things like logs, backups, etc. You can apply similar concepts to individual resources more granularly than the account. Disclaimer: principal at AWS but the above is my own opinion/observation and does not represent my employer.