6 ms·

Lesson for those designing cold-storage solutions--design your system, as much as reasonable, to not support deletion operations. The reason why files get lost

Lesson for those designing cold-storage solutions--design your system, as much as reasonable, to not support deletion operations. The reason why files get lost is usually due to software bugs, bad configurations, and operator errors. Design your system to protect against these things.

_Microft 6y ago
This reminds me of this comment that was posted in the "medical data leak" discussion that is currently on the front page as well: https://news.ycombinator.com/item?id=24196041 https://news.ycombinator.com/item?id=24196041
CalRobert 6y ago
If memory serves, a class of Glacier storage (AWS) isn't practical to use in the EU because the inability to delete makes it impossible to serve "Right to Erasure" GDPR requests.
- londons_explore 6y ago
  Store data and keys seperately. The set of keys for all your customers should be only a few megabytes, so is much much easier to back up more frequently and more copies of. It's also easier to have a "we do daily backups and delete any backups after 21 days" policy.
  - lozenge 6y ago
    Storing names separately can help but the subset of data you have a legitimate legal reason to "refuse to forget" can change faster than you would want to rewrite your backups(well, nobody wants to rewrite backups, but you get what I mean)
    - raverbashing 6y ago
      You need to denormalize the non-forgettable information. This way you don't have to worry about what you can delete or not. Store Invoices, billing, etc, separately. These also usually have a fixed pre-determined retaining policy time. Past that time? Anonymize/aggregate or just delete it.
      - soco 6y ago
        Legislation can change and what you once thought to be forgettable becomes suddenly not.
        aaronmdjones 6y ago
        It'd be extremely unusual for such changes to be retrospective. For example, if you are currently required to keep certain kinds of financial records for 3 years, and you delete records older than that, you won't be held at fault when that law gets updated to say 5 years, and you then fail to produce a 4-year-old document; it has not yet been a year since the new law took effect. This extends to criminal law too, for example with statutes of limitation. If, 20 years and 1 day ago, there was a 20-year statute of limitations for murder charges, you cannot be charged for committing a murder on that day (barring extraneous circumstances, such as tolling); even if a law was passed the day after said murder removing this limitation.
        soco 6y ago
        Well, yes, but the topic here was data. And if at some time you thought it was forgettable and you scheduled it for deletion, once the law changes you should make damn sure all code involved in that deletion was checked or the data will be gone - against the new law. A big opportunity for human error here.
  - syrgian 6y ago
    Wouldn't you need to edit every single backup where the key is stored in order to comply?
    - dodobirdlord 6y ago
      I assume they mean the standard practice of encrypting your backups with per-customer encryption keys. If you ever want to destroy all of the data for a particular customer you don't have to do any editing of backups, you just have to nuke every copy of the encryption key you use for that customer.
- Hamuko 6y ago
  Not necessarily. While it's not set in stone, GDPR right to erasure might not be in conflict with backup retention. >The GDPR is open to interpretation, so we asked an EU Member State supervisory authority (CNIL in France) for clarification. CNIL confirmed that you’ll have one month to answer to a removal request, and that you don’t need to delete a backup set in order to remove an individual from it. Organizations will have to clearly explain to the data subject (using clear and plain language) that his or her personal data has been removed from production systems, but a backup copy may remain, but will expire after a certain amount of time (indicate the retention time in your communication with the data subject). Backups should only be used for restoring a technical environment, and data subject personal data should not be processed again after restore (and deleted again). While this adds some complexity, it allows organizations to have some time to re-engineer their data protection processes. https://blog.quantum.com/2018/01/26/backup-administrators-the-1-advice-to-deal-with-gdpr-and-the-right-of-erasure/ https://blog.quantum.com/2018/01/26/backup-administrators-th...
  - johnchristopher 6y ago
    As I understood it at the time this is the difference between archives and backup.
- eru 6y ago
  I suspect you can square the circle, if you encrypt all the files, and store the keys in storage you can delete quickly.
- CalRobert 6y ago
  I see memory served me incorrectly! Apologies if I have spread misinformation. It was an offhand comment from an AWS rep a few years ago and things may have changed.
  - klodolph 6y ago
    When GDPR passed there was a bit of a scramble to understand it, and people had all sorts of different interpretations that they were sharing.
Sharlin 6y ago
To be fair, image.canon is definitely not a cold-storage or backup solution, never mind primary storage. Its main use case is acting as a buffer for auto-uploading photos from camera and then later downloading them to a computer. As an afterthought it also offers a tiny long-term storage space, 10 GB per user, smaller than most memory cards available today.
- SifJar 6y ago
  It also appears that only images stored for >30 days were affected, so in theory most photos "should" have been downloaded and stored properly etc. if people are using the service as you describe
stingraycharles 6y ago
Yeah, it seems some of the other commenters miss that point: it’s not as if some disk crashed or something, it’s a bug in a software upgrade that deleted long term storage in addition to short term storage. In my experience, the best way to mitigate these kind of things is to use PITR, which is always immutable. You make periodic snapshots, keep track of all changes since those snapshots. And then you use a policy to determine just how much data you want to keep here. The problem is that not supporting delete operations is often very much impractical. Consider for example the recent heat Flickr (or was it Instagram?) has gotten for not physically deleting files but just marking them as deleted/invisible. You sometimes have really good reasons that you must support delete operations, so you must just make sure that there’s a reasonable delay before they are actually completely gone.
- tonyedgecombe 6y ago
  In case anybody else wonders what PITR is: https://en.wikipedia.org/wiki/Point-in-time_recovery https://en.wikipedia.org/wiki/Point-in-time_recovery
mschuster91 6y ago
The problem with that is that under GDPR you have to support full deletion of an user's data.
- Hamuko 6y ago
  Not necessarily if you have a valid reason to storing the data. You might for example have to keep user's billing data around for legal reasons.
  - NullPrefix 6y ago
    Was this billing data though?
- toong 6y ago
  Store customer data immutable and encrypted, with a unique AES encryption key per customer. Encrypt that AES key again with a single RSA key-pair and store the encrypted AES-key in a database. You can access your customer data, using the customer-specific AES key. You can access the customer-specific AES key using your private RSA key. When you need to delete the customer data under GDPR, you can delete the encrypted AES key for that customer from your database.
  - chmod775 6y ago
    > Store customer data immutable and encrypted, with a unique AES encryption key per customer. Encrypt that AES key again with a single RSA key-pair and store the encrypted AES-key in a database. Now you have the worst of both worlds. You also now have 2 points of failure where data can get lost, because if either has a problem you lose data.
    - Ensorceled 6y ago
      I can't think of a mechanism where you can lose the customer's database records but can still recover files associated with that customer ...
mixedbit 6y ago
Both AWS S3 and Google cloud storage buckets have an option that makes it impossible to delete stored objects for some period of time. The option was added for some legal compliance reasons, but I find it useful as an extra safeguard that important service data is not accidentally or maliciously deleted.
- ramraj07 6y ago
  You could always not pay the bill :)
- chrismatheson 6y ago
  I suppose that feature would be incompatible with user data in a GDPR world ??
  - visarga 6y ago
    Git is also incompatible with GDPR, you can't simply delete a file from all history.
    - nix23 6y ago
      Oh man, all my WORM tapes are not GDPR compatible...thats terrible!!!
    - icebraining 6y ago
      You can, you just have to be willing to rewrite history. See git-filter-branch (and don't forget to read the fat WARNING).
    - msh 6y ago
      Git is not really intended for personal data
      - Hamuko 6y ago
        Yeah, I don't really understand why you would like to stuff personal data into a git repository (unless we're strictly speaking about author data). It's really not the tool for it. Now for those who decided that a blockchain is the perfect solution for storing personal data however...
lmilcin 6y ago
This is actually larger lesson. When I work on design I try to make catastrophic failure impossible. Not implementing deletion or implementing deletion with some unseparable constraint check is one way. For example, the files could have a flag set and from that point on the only delete function would check the flag and refuse to remove the image regardless of how it got invoked or where the image is placed (think like write protect notch on a floppy disk). Another way is to not actually remove the files but instead introduce grace period when they are marked as deleted, not accessible, to the user. The only real deletion would happen in a vacuuming system that will actually remove the files but with a hardcoded constraint next to filesystem delet that the file must have been marked as removed at least XX days ago. This would give some time after deletion to actually recover files. Yet another technique is to move objects marked as deleted to some other, cheaper form of storage. The function that moves objects must first confirm the object is available in new storage through its API and only then removes the original. You can also prevent damage that could result from failed modification using immutable objects and Copy on Write. The copies would be deleted using the above described mechanism. Yet another technique is to keep XX days of redo-log that could contain enough information to rebuild any object to any of its current or past state within XX days. Depending on the type of the file and nature of changes it can be much more efficient than keeping whole copies. All of this of course adds overhead but you can recover a lot of that overhead by relaxing requirements on duplication of your operational or other forms of backup storage with rationale that if you have a redo log where you can locate current or past version of the object independently of operational or backup storage you can count it as one more operational or backup copy.
IshKebab 6y ago
Good luck complying with the GDPA.
- klodolph 6y ago
  Do you mean GDPR? You don’t have to delete data immediately under the right to be forgotten.

_Microft 6y ago

This reminds me of this comment that was posted in the "medical data leak" discussion that is currently on the front page as well: https://news.ycombinator.com/item?id=24196041 https://news.ycombinator.com/item?id=24196041

CalRobert 6y ago

If memory serves, a class of Glacier storage (AWS) isn't practical to use in the EU because the inability to delete makes it impossible to serve "Right to Erasure" GDPR requests.

londons_explore 6y ago
Store data and keys seperately. The set of keys for all your customers should be only a few megabytes, so is much much easier to back up more frequently and more copies of. It's also easier to have a "we do daily backups and delete any backups after 21 days" policy.
- lozenge 6y ago
  Storing names separately can help but the subset of data you have a legitimate legal reason to "refuse to forget" can change faster than you would want to rewrite your backups(well, nobody wants to rewrite backups, but you get what I mean)
  - raverbashing 6y ago
    You need to denormalize the non-forgettable information. This way you don't have to worry about what you can delete or not. Store Invoices, billing, etc, separately. These also usually have a fixed pre-determined retaining policy time. Past that time? Anonymize/aggregate or just delete it.
    - soco 6y ago
      Legislation can change and what you once thought to be forgettable becomes suddenly not.
      - aaronmdjones 6y ago
        It'd be extremely unusual for such changes to be retrospective. For example, if you are currently required to keep certain kinds of financial records for 3 years, and you delete records older than that, you won't be held at fault when that law gets updated to say 5 years, and you then fail to produce a 4-year-old document; it has not yet been a year since the new law took effect. This extends to criminal law too, for example with statutes of limitation. If, 20 years and 1 day ago, there was a 20-year statute of limitations for murder charges, you cannot be charged for committing a murder on that day (barring extraneous circumstances, such as tolling); even if a law was passed the day after said murder removing this limitation.
        soco 6y ago
        Well, yes, but the topic here was data. And if at some time you thought it was forgettable and you scheduled it for deletion, once the law changes you should make damn sure all code involved in that deletion was checked or the data will be gone - against the new law. A big opportunity for human error here.
- syrgian 6y ago
  Wouldn't you need to edit every single backup where the key is stored in order to comply?
  - dodobirdlord 6y ago
    I assume they mean the standard practice of encrypting your backups with per-customer encryption keys. If you ever want to destroy all of the data for a particular customer you don't have to do any editing of backups, you just have to nuke every copy of the encryption key you use for that customer.
Hamuko 6y ago
Not necessarily. While it's not set in stone, GDPR right to erasure might not be in conflict with backup retention. >The GDPR is open to interpretation, so we asked an EU Member State supervisory authority (CNIL in France) for clarification. CNIL confirmed that you’ll have one month to answer to a removal request, and that you don’t need to delete a backup set in order to remove an individual from it. Organizations will have to clearly explain to the data subject (using clear and plain language) that his or her personal data has been removed from production systems, but a backup copy may remain, but will expire after a certain amount of time (indicate the retention time in your communication with the data subject). Backups should only be used for restoring a technical environment, and data subject personal data should not be processed again after restore (and deleted again). While this adds some complexity, it allows organizations to have some time to re-engineer their data protection processes. https://blog.quantum.com/2018/01/26/backup-administrators-the-1-advice-to-deal-with-gdpr-and-the-right-of-erasure/ https://blog.quantum.com/2018/01/26/backup-administrators-th...
- johnchristopher 6y ago
  As I understood it at the time this is the difference between archives and backup.
eru 6y ago
I suspect you can square the circle, if you encrypt all the files, and store the keys in storage you can delete quickly.
CalRobert 6y ago
I see memory served me incorrectly! Apologies if I have spread misinformation. It was an offhand comment from an AWS rep a few years ago and things may have changed.
- klodolph 6y ago
  When GDPR passed there was a bit of a scramble to understand it, and people had all sorts of different interpretations that they were sharing.

Sharlin 6y ago

To be fair, image.canon is definitely not a cold-storage or backup solution, never mind primary storage. Its main use case is acting as a buffer for auto-uploading photos from camera and then later downloading them to a computer. As an afterthought it also offers a tiny long-term storage space, 10 GB per user, smaller than most memory cards available today.

SifJar 6y ago
It also appears that only images stored for >30 days were affected, so in theory most photos "should" have been downloaded and stored properly etc. if people are using the service as you describe

stingraycharles 6y ago

Yeah, it seems some of the other commenters miss that point: it’s not as if some disk crashed or something, it’s a bug in a software upgrade that deleted long term storage in addition to short term storage. In my experience, the best way to mitigate these kind of things is to use PITR, which is always immutable. You make periodic snapshots, keep track of all changes since those snapshots. And then you use a policy to determine just how much data you want to keep here. The problem is that not supporting delete operations is often very much impractical. Consider for example the recent heat Flickr (or was it Instagram?) has gotten for not physically deleting files but just marking them as deleted/invisible. You sometimes have really good reasons that you must support delete operations, so you must just make sure that there’s a reasonable delay before they are actually completely gone.

tonyedgecombe 6y ago
In case anybody else wonders what PITR is: https://en.wikipedia.org/wiki/Point-in-time_recovery https://en.wikipedia.org/wiki/Point-in-time_recovery

mschuster91 6y ago

The problem with that is that under GDPR you have to support full deletion of an user's data.

Hamuko 6y ago
Not necessarily if you have a valid reason to storing the data. You might for example have to keep user's billing data around for legal reasons.
- NullPrefix 6y ago
  Was this billing data though?
toong 6y ago
Store customer data immutable and encrypted, with a unique AES encryption key per customer. Encrypt that AES key again with a single RSA key-pair and store the encrypted AES-key in a database. You can access your customer data, using the customer-specific AES key. You can access the customer-specific AES key using your private RSA key. When you need to delete the customer data under GDPR, you can delete the encrypted AES key for that customer from your database.
- chmod775 6y ago
  > Store customer data immutable and encrypted, with a unique AES encryption key per customer. Encrypt that AES key again with a single RSA key-pair and store the encrypted AES-key in a database. Now you have the worst of both worlds. You also now have 2 points of failure where data can get lost, because if either has a problem you lose data.
  - Ensorceled 6y ago
    I can't think of a mechanism where you can lose the customer's database records but can still recover files associated with that customer ...

mixedbit 6y ago

Both AWS S3 and Google cloud storage buckets have an option that makes it impossible to delete stored objects for some period of time. The option was added for some legal compliance reasons, but I find it useful as an extra safeguard that important service data is not accidentally or maliciously deleted.

ramraj07 6y ago
You could always not pay the bill :)
chrismatheson 6y ago
I suppose that feature would be incompatible with user data in a GDPR world ??
- visarga 6y ago
  Git is also incompatible with GDPR, you can't simply delete a file from all history.
  - nix23 6y ago
    Oh man, all my WORM tapes are not GDPR compatible...thats terrible!!!
  - icebraining 6y ago
    You can, you just have to be willing to rewrite history. See git-filter-branch (and don't forget to read the fat WARNING).
  - msh 6y ago
    Git is not really intended for personal data
    - Hamuko 6y ago
      Yeah, I don't really understand why you would like to stuff personal data into a git repository (unless we're strictly speaking about author data). It's really not the tool for it. Now for those who decided that a blockchain is the perfect solution for storing personal data however...

lmilcin 6y ago

This is actually larger lesson. When I work on design I try to make catastrophic failure impossible. Not implementing deletion or implementing deletion with some unseparable constraint check is one way. For example, the files could have a flag set and from that point on the only delete function would check the flag and refuse to remove the image regardless of how it got invoked or where the image is placed (think like write protect notch on a floppy disk). Another way is to not actually remove the files but instead introduce grace period when they are marked as deleted, not accessible, to the user. The only real deletion would happen in a vacuuming system that will actually remove the files but with a hardcoded constraint next to filesystem delet that the file must have been marked as removed at least XX days ago. This would give some time after deletion to actually recover files. Yet another technique is to move objects marked as deleted to some other, cheaper form of storage. The function that moves objects must first confirm the object is available in new storage through its API and only then removes the original. You can also prevent damage that could result from failed modification using immutable objects and Copy on Write. The copies would be deleted using the above described mechanism. Yet another technique is to keep XX days of redo-log that could contain enough information to rebuild any object to any of its current or past state within XX days. Depending on the type of the file and nature of changes it can be much more efficient than keeping whole copies. All of this of course adds overhead but you can recover a lot of that overhead by relaxing requirements on duplication of your operational or other forms of backup storage with rationale that if you have a redo log where you can locate current or past version of the object independently of operational or backup storage you can count it as one more operational or backup copy.

IshKebab 6y ago

Good luck complying with the GDPA.

klodolph 6y ago
Do you mean GDPR? You don’t have to delete data immediately under the right to be forgotten.