5 ms·

Now that storage is infinite and cheap, it's become trendy for engineers to never actually remove anything, but to just set a "deleted" flag in a database somew

by ohazi 6y ago

Now that storage is infinite and cheap, it's become trendy for engineers to never actually remove anything, but to just set a "deleted" flag in a database somewhere. This makes a lot of tricky things easier, but there are pitfalls. Years later, in the fifteenth re-implementation of some part of some service, somebody is going to forget to read that bit. Or an errant database migration isn't going to transfer it properly. Or the database will be leaked, and all of the "deleted" content will still be there (e.g. the Parler dump). Or a race condition will cause the check to occasionally be ignored every other Sunday. Discovering that this is happening can be next to impossible. An application might read the deleted bit correctly when generating the ordered list of stuff you've looked at, but might not read it correctly (or might deliberately ignore it) when generating your advertising profile or when making content recommendations. I swear I've seen this on Amazon, though of course, I'll never be able to prove it. Chilling effects / social cooling includes stuff like this. "I'm not even going to click on that, because I don't trust the platform to clear my history properly."

genghizkhan 6y ago
Just look at Parler. That's the same thing which happened there.
debaserab2 6y ago
Soft deletes have been popular since the internet became accessible to the average person. When I was younger I was a huge advocate for soft deleting pretty much everything. It felt like the obvious, easy choice - why not retain the data incase you need to use it for something else down the road? It took years to understand the ramifications of storing that data in a secure way really means, and that often times the reason it gets used later isn't something you intended it to be used for when you designed the data structure. I don't think that way anymore and I truly believe hard deletes should be the default unless you have a very compelling reason to soft delete (and you should usually only soft delete with some guarantee of future deletion via a publicly accessible data retention/erasure policy)
- 05 6y ago
  It's not rocket science - encrypt with a one time symmetric key, encrypt the key with a public key, store blob and key marked as deleted. Store the corresponding private asymmetric key on a hardware token to be used when you decide you need the deleted data. That way no db hack exposes any usable data.
  - debaserab2 6y ago
    DB hacks aren't even close to being the only (or worst) threat. Encryption means nothing in a company or bankruptcy buyout where the purchaser may have a completely different set of ethics than yours.
  - WJW 6y ago
    That is all good and well against simple actors, but does not defend against the company being bought out and the new owner(s) deciding to decrypt and sell the entire database to some shady data broker. Or against the FBI/your local equivalent coming over for a friendly visit and confiscating both the hardware token and the database.
dexen 6y ago
>This makes a lot of tricky things easier, but there are pitfalls. Some people are calling excessive data "toxic asset", due to the future risks of it causing unexpected breaches of privacy, damage to reputation, or other losses. We'll certainly develop standard technical means of handling that in due time, quite possibly ones based on cryptography. A common scenario at present: instead of trawling all backups & distributed stores to laboriously expunge every bit of deleted data, you could delete the relevant encryption keys - a small piece of data, a known quantity - rendering the encrypted data inert. This is particularly suited for systems that use WORM approach to backing up & distributing data.
deckard1 6y ago
> just set a "deleted" flag in a database somewhere this has been a thing since the 1980s, at least. It's not new, and it's often important for auditing purposes and because mistakes happen. What's new is the ease with which databases can be accessed and hacked over a global public network. > Or an errant database migration isn't going to transfer it properly. Or the database will be leaked, and all of the "deleted" content will still be there Backups are a thing, you know. When you issue a delete command to your database it's not reaching back to time immemorial through the archives and ripping out those records. If we're playing what-ifs, then backup databases can just as easily be leaked.
globular-toast 6y ago
It's not trendy, it's how I've been doing databases my whole life. You don't let users actually delete stuff from your database. That's crazy.
- Lio 6y ago
  How would you handle a user ceasing their account and asking you to remove data you no longer need to provide them with service? Under GDPR you'd be required to really delete that. Obviously GDPR laws don't hold everywhere but are you saying you'd just hold the user's data forever or that you'd have a later clean up process or something?
  - globular-toast 6y ago
    You wouldn't. GDPR didn't exist. That was my point. We've been doing this for ages and only recently had to worry about actually deleting stuff. This is why I've never advocated "deleting" Facebook accounts etc. If you "delete" it you're just giving them another piece of information about you. Namely that you want your account to be deleted. The GDPR has not yet been shown to have any teeth, so I assume nothing has really changed in this respect. And, yes, holding users' data forever is normal. Storage cost has only decreased relative to the size of these databases. I've worked in places with customer data going back decades.
theshrike79 6y ago
> Now that storage is infinite and cheap, it's become trendy for engineers to never actually remove anything, but to just set a "deleted" flag in a database somewhere. This makes a lot of tricky things easier, but there are pitfalls. Also deleting stuff for real for real immediately is really hard. You need to delete it from the database, the one that's replicating to, caches, online backups AND offline backups. And that's after you've confirmed and reconfirmed from the user that they actually want to delete the data, not just store it in the recycle bin or something idiotic like that. The easiest way is just to flag as deleted and prune on some kind of schedule. Backups will be overwritten at some point and the data will go away eventually.
- TeMPOraL 6y ago
  Or, though a more technically complex, encrypt every "deletable" unit of data with its own key, have a different backup policy for encrypted data and for keys to it - one which, upon user request, would allow you to quickly purge every trace of relevant keys.
  - jandrewrogers 6y ago
    This doesn't work well in practice at any kind of scale -- these types of databases have existed for many years. The key management overhead is considerable, causing integer factor loss of performance for database-y systems. Additionally, it very significantly increases the storage footprint since most techniques for minimizing storage utilization no longer work given this requirement. It isn't just "technically complex", the limiting factor is your system may be 10x slower and use 10x more storage. The economics of operating these systems is so poor that they are only used in extremely niche environments where the requirements justify the extreme cost and performance limitations.
parliament32 6y ago
This is exactly what happened with the AshleyMadison hack. In fact, users explicitly paid for a "delete my account permanently" action.. yet it was just that, a "deleted" flag added to the DB which eventually got leaked.

by ohazi 6y ago

genghizkhan 6y ago

Just look at Parler. That's the same thing which happened there.

debaserab2 6y ago

Soft deletes have been popular since the internet became accessible to the average person. When I was younger I was a huge advocate for soft deleting pretty much everything. It felt like the obvious, easy choice - why not retain the data incase you need to use it for something else down the road? It took years to understand the ramifications of storing that data in a secure way really means, and that often times the reason it gets used later isn't something you intended it to be used for when you designed the data structure. I don't think that way anymore and I truly believe hard deletes should be the default unless you have a very compelling reason to soft delete (and you should usually only soft delete with some guarantee of future deletion via a publicly accessible data retention/erasure policy)

05 6y ago
It's not rocket science - encrypt with a one time symmetric key, encrypt the key with a public key, store blob and key marked as deleted. Store the corresponding private asymmetric key on a hardware token to be used when you decide you need the deleted data. That way no db hack exposes any usable data.
- debaserab2 6y ago
  DB hacks aren't even close to being the only (or worst) threat. Encryption means nothing in a company or bankruptcy buyout where the purchaser may have a completely different set of ethics than yours.
- WJW 6y ago
  That is all good and well against simple actors, but does not defend against the company being bought out and the new owner(s) deciding to decrypt and sell the entire database to some shady data broker. Or against the FBI/your local equivalent coming over for a friendly visit and confiscating both the hardware token and the database.

dexen 6y ago

>This makes a lot of tricky things easier, but there are pitfalls. Some people are calling excessive data "toxic asset", due to the future risks of it causing unexpected breaches of privacy, damage to reputation, or other losses. We'll certainly develop standard technical means of handling that in due time, quite possibly ones based on cryptography. A common scenario at present: instead of trawling all backups & distributed stores to laboriously expunge every bit of deleted data, you could delete the relevant encryption keys - a small piece of data, a known quantity - rendering the encrypted data inert. This is particularly suited for systems that use WORM approach to backing up & distributing data.

deckard1 6y ago

> just set a "deleted" flag in a database somewhere this has been a thing since the 1980s, at least. It's not new, and it's often important for auditing purposes and because mistakes happen. What's new is the ease with which databases can be accessed and hacked over a global public network. > Or an errant database migration isn't going to transfer it properly. Or the database will be leaked, and all of the "deleted" content will still be there Backups are a thing, you know. When you issue a delete command to your database it's not reaching back to time immemorial through the archives and ripping out those records. If we're playing what-ifs, then backup databases can just as easily be leaked.

globular-toast 6y ago

It's not trendy, it's how I've been doing databases my whole life. You don't let users actually delete stuff from your database. That's crazy.

Lio 6y ago
How would you handle a user ceasing their account and asking you to remove data you no longer need to provide them with service? Under GDPR you'd be required to really delete that. Obviously GDPR laws don't hold everywhere but are you saying you'd just hold the user's data forever or that you'd have a later clean up process or something?
- globular-toast 6y ago
  You wouldn't. GDPR didn't exist. That was my point. We've been doing this for ages and only recently had to worry about actually deleting stuff. This is why I've never advocated "deleting" Facebook accounts etc. If you "delete" it you're just giving them another piece of information about you. Namely that you want your account to be deleted. The GDPR has not yet been shown to have any teeth, so I assume nothing has really changed in this respect. And, yes, holding users' data forever is normal. Storage cost has only decreased relative to the size of these databases. I've worked in places with customer data going back decades.

theshrike79 6y ago

> Now that storage is infinite and cheap, it's become trendy for engineers to never actually remove anything, but to just set a "deleted" flag in a database somewhere. This makes a lot of tricky things easier, but there are pitfalls. Also deleting stuff for real for real immediately is really hard. You need to delete it from the database, the one that's replicating to, caches, online backups AND offline backups. And that's after you've confirmed and reconfirmed from the user that they actually want to delete the data, not just store it in the recycle bin or something idiotic like that. The easiest way is just to flag as deleted and prune on some kind of schedule. Backups will be overwritten at some point and the data will go away eventually.

TeMPOraL 6y ago
Or, though a more technically complex, encrypt every "deletable" unit of data with its own key, have a different backup policy for encrypted data and for keys to it - one which, upon user request, would allow you to quickly purge every trace of relevant keys.
- jandrewrogers 6y ago
  This doesn't work well in practice at any kind of scale -- these types of databases have existed for many years. The key management overhead is considerable, causing integer factor loss of performance for database-y systems. Additionally, it very significantly increases the storage footprint since most techniques for minimizing storage utilization no longer work given this requirement. It isn't just "technically complex", the limiting factor is your system may be 10x slower and use 10x more storage. The economics of operating these systems is so poor that they are only used in extremely niche environments where the requirements justify the extreme cost and performance limitations.

parliament32 6y ago

This is exactly what happened with the AshleyMadison hack. In fact, users explicitly paid for a "delete my account permanently" action.. yet it was just that, a "deleted" flag added to the DB which eventually got leaked.