6 ms·

I do wonder whether the data is actually deleted or they just do like every other large tech company and set a “deleted = true” flag that hides it from the user

I do wonder whether the data is actually deleted or they just do like every other large tech company and set a “deleted = true” flag that hides it from the user, since actually deleting it and all of its replicates across data centres, backup images, and tape storage is from what I’ve been told, virtually impossible even if the will was there.

kshacker 2y ago
Our industry's dark secrets getting exposed like this ... yeah you are right, we don't even need a whistleblower for this :)
bbor 2y ago
It is the law in California, and it is certainly possible at scale, just not cheap or trivial. Presumably, given the intense focus of the state government on their company in particular, they have long since implemented a compliant deletion mechanism! The question, of course, is whether that same functionality is applied to residents of other states wishing to delete their data…
someothherguyy 2y ago
https://en.wiktionary.org/wiki/soft_deletion https://en.wiktionary.org/wiki/soft_deletion
- silisili 2y ago
  It's absurd to me how soft deletion took over the entire industry and 'DELETE FROM' became taboo.
  - raverbashing 2y ago
    Too many fat fingers, too many relations that were not suppose to break (or were supposed to be disjoint but weren't), too much regulatory uncertainty, both for deleting and keeping the data, etc etc
    - silisili 2y ago
      The regulatory aspect is one I get. The problem is that it infected the mindset of almost every company. Those that worked at companies who needed it brought the mindset everywhere. And like many technologies, people who study what big companies do mimic it everywhere, even where it's not needed. So your simple stupid SaaS company has some simple object that a user wants to delete that ends up living in a db somewhere... forever.
  - msie 2y ago
    Not really, soft deletion has saved many people's bacon.
    - al_borland 2y ago
      That doesn’t mean soft delete should be the only option. With some of our processes, we soft delete for a period of time (2-4 weeks) to account for mistakes. Then, if on one tries to recover it after that time, it does the actual delete. This seems like a reasonable compromise to me. It offers a safety net, while still getting the job done. With as many data breaches as we see these days, I want my data gone. If someone gets a dump of the DB, a delete flag doesn’t matter. This was the primary reason I never got a DNA test, even though I want to see what my results would be. I never trusted the companies to store that data long term. If I could get a test where a company didn’t store the results after they were provided to me, I’d get one tomorrow.
      - johnisgood 2y ago
        The AT&T leak was huge. :D There was also a leak in either 23andMe or some other company handling DNAs not too long ago (months, maybe 2 years >.>).
  - friendzis 2y ago
    To `DELETE FROM` you have to have at least a relatively vague idea what you are doing. Go and find an organization that has their data and component dependency maps ready and up to date :) One hard delete can expose months worth of bugs.
    - silisili 2y ago
      'Our data design is too bad to allow anyone to delete anything' is not, to me, a valid reason to prevent actual deletion.
      - groestl 2y ago
        Enter Git. (FTR, I don't think it's bad data design. It's a tradeoff. Sometimes it's even a feature.)
    - MrMcCall 2y ago
      Can't delete from backups, though.
      - friendzis 2y ago
        If you really cannot (although, most probably just burdensome), you can always render the data unreadable. E.g. by destroying the keys.
        beejiu 2y ago
        Don't forget to backup those keys, though :)
  - goosejuice 2y ago
    For every customer that legitimately wants their data deleted there's a hundred that complain their data is gone after confirming deletion. The same customers that mark your transactional email as spam then complain about not being able to get into their account. Exaggerating here, but I do think soft deletion practices can be partially blamed on real business problems. I'm not sure why anyone would think that using soft deletion on an actual burn account function would be sensible though.
    - silisili 2y ago
      I get that a little from the company perspective, but it leads to its own issues of perhaps more important magnitude. If you have data for a user, who wants to delete it, you can soft delete to allow recovery to prevent complaints. On the other hand, 5 years from now your system gets hacked, and now you have to email some person you haven't done business with in 5 years to tell them their data that you claimed was deleted was leaked. Sure you can do soft deletions with some hacky actual delete schedule but it doesn't really solve the problem, only reduces the timeline.
      - johnisgood 2y ago
        Why not educate the customer? "If you request deletion, do not expect your data back, because it truly will be deleted!". Sad state of world that this has to be said, however. Discord (and many other places) allow you to recover your account within a specific period of time, but they do not delete anything, you just simply lose access and your name gets changed (and avatar and status removed), but all your DMs are there, all your messages are there (although I understand why, especially in servers (guilds)). There is a way to bulk save and delete messages from Discord, however.
        speff 2y ago
        People in general simply do not read warnings like that. It’s a form of autopilot that everyone utilizes to function/cope in an age where they’re being bombarded with too much information
  - nickm12 2y ago
    Says someone who has never experienced a data loss bug.
    - silisili 2y ago
      Was the data really important? Is it worth holding people's data hostage because of subpar developers? I'm not at all saying that soft deletion shouldn't exist. I'm arguing that hard deletion should be the default, and then soft deletion practices should have some justification. Instead the industry chose to just soft delete everywhere, which is good for the industry, and bad for users.
      - fc417fc802 2y ago
        Soft delete is a sensible default to prevent all sorts of problems. It's almost always good for a human to be able to get into the system and revert things if something bad happened. It should just be followed up by hard delete several days or weeks later. The general policy should never be to hold things indefinitely. As a practical example I have soft deletion for more or less everything on my desktop. There are periodic filesystem level snapshots and those stick around until I manually GC them. It has saved me more than once.
      - nickm12 1y ago
        The industry chose the default of "data that is stored in durable storage is probably important". Sorry, I just don't see how that is bad for users when compared to the alternative default of "data that is stored in durable storage should be irretrievably deleted immediately".
  - groestl 2y ago
    Not deleting something you should have: a fine. In some countries. Deleting something you should not have: potentially a business extinction event. And a fine as well.
  - timeon 2y ago
    Seems risky to put your data on non-EU servers. It is still risk, but at least there is legal framework.
carimura 2y ago
it very likely may not be per their own words. They are "bound by various legal and regulatory obligations that may necessitate retention of certain information". This came straight from their privacy team.
Unroasted6154 2y ago
It possible with crypto shredding. You store everything encrypted with one key by customer. When you want to delete you erase the key. The data becomes unusable everywhere (backups included). Then a job periodically garbage collects data without a key, but that's more for cost saving. Big companies do this but it requires some technical maturity. If you operate in Europe you have to implement proper data deletion. I would be more worried about small companies that large ones tombe honest.
- Quekid5 2y ago
  You still have to backup those keys somewhere... and if you don't do it the same way as for the data then your backups are effectively worthless.
  - ripped_britches 2y ago
    Yea was going to say this. I would be surprised if any major company does this for normal uses.
    - Unroasted6154 2y ago
      That's how GAFAMs size companies do it. They need to comply with European regulations for data deletion, and there are very few other options.
  - muti 2y ago
    Much less data to back up so it can be stored in a way that is replicated for redundancy but still mutable. Separating the key and data is what allows for sending data to tape backup etc
    - Quekid5 2y ago
      If your (backup-via-redundancy) keys are mutable, you do not have a backup. What happens in the case of a ransomware attack, for example? You've also added (possibly substantial) latency to every single operation that operates on user data.
tdeck 2y ago
Soft deletion doesn't satisfy the company's obligation under the CCPA.

kshacker 2y ago

Our industry's dark secrets getting exposed like this ... yeah you are right, we don't even need a whistleblower for this :)

bbor 2y ago

It is the law in California, and it is certainly possible at scale, just not cheap or trivial. Presumably, given the intense focus of the state government on their company in particular, they have long since implemented a compliant deletion mechanism! The question, of course, is whether that same functionality is applied to residents of other states wishing to delete their data…

someothherguyy 2y ago

https://en.wiktionary.org/wiki/soft_deletion https://en.wiktionary.org/wiki/soft_deletion

silisili 2y ago
It's absurd to me how soft deletion took over the entire industry and 'DELETE FROM' became taboo.
- raverbashing 2y ago
  Too many fat fingers, too many relations that were not suppose to break (or were supposed to be disjoint but weren't), too much regulatory uncertainty, both for deleting and keeping the data, etc etc
  - silisili 2y ago
    The regulatory aspect is one I get. The problem is that it infected the mindset of almost every company. Those that worked at companies who needed it brought the mindset everywhere. And like many technologies, people who study what big companies do mimic it everywhere, even where it's not needed. So your simple stupid SaaS company has some simple object that a user wants to delete that ends up living in a db somewhere... forever.
- msie 2y ago
  Not really, soft deletion has saved many people's bacon.
  - al_borland 2y ago
    That doesn’t mean soft delete should be the only option. With some of our processes, we soft delete for a period of time (2-4 weeks) to account for mistakes. Then, if on one tries to recover it after that time, it does the actual delete. This seems like a reasonable compromise to me. It offers a safety net, while still getting the job done. With as many data breaches as we see these days, I want my data gone. If someone gets a dump of the DB, a delete flag doesn’t matter. This was the primary reason I never got a DNA test, even though I want to see what my results would be. I never trusted the companies to store that data long term. If I could get a test where a company didn’t store the results after they were provided to me, I’d get one tomorrow.
    - johnisgood 2y ago
      The AT&T leak was huge. :D There was also a leak in either 23andMe or some other company handling DNAs not too long ago (months, maybe 2 years >.>).
- friendzis 2y ago
  To `DELETE FROM` you have to have at least a relatively vague idea what you are doing. Go and find an organization that has their data and component dependency maps ready and up to date :) One hard delete can expose months worth of bugs.
  - silisili 2y ago
    'Our data design is too bad to allow anyone to delete anything' is not, to me, a valid reason to prevent actual deletion.
    - groestl 2y ago
      Enter Git. (FTR, I don't think it's bad data design. It's a tradeoff. Sometimes it's even a feature.)
  - MrMcCall 2y ago
    Can't delete from backups, though.
    - friendzis 2y ago
      If you really cannot (although, most probably just burdensome), you can always render the data unreadable. E.g. by destroying the keys.
      - beejiu 2y ago
        Don't forget to backup those keys, though :)
- goosejuice 2y ago
  For every customer that legitimately wants their data deleted there's a hundred that complain their data is gone after confirming deletion. The same customers that mark your transactional email as spam then complain about not being able to get into their account. Exaggerating here, but I do think soft deletion practices can be partially blamed on real business problems. I'm not sure why anyone would think that using soft deletion on an actual burn account function would be sensible though.
  - silisili 2y ago
    I get that a little from the company perspective, but it leads to its own issues of perhaps more important magnitude. If you have data for a user, who wants to delete it, you can soft delete to allow recovery to prevent complaints. On the other hand, 5 years from now your system gets hacked, and now you have to email some person you haven't done business with in 5 years to tell them their data that you claimed was deleted was leaked. Sure you can do soft deletions with some hacky actual delete schedule but it doesn't really solve the problem, only reduces the timeline.
    - johnisgood 2y ago
      Why not educate the customer? "If you request deletion, do not expect your data back, because it truly will be deleted!". Sad state of world that this has to be said, however. Discord (and many other places) allow you to recover your account within a specific period of time, but they do not delete anything, you just simply lose access and your name gets changed (and avatar and status removed), but all your DMs are there, all your messages are there (although I understand why, especially in servers (guilds)). There is a way to bulk save and delete messages from Discord, however.
      - speff 2y ago
        People in general simply do not read warnings like that. It’s a form of autopilot that everyone utilizes to function/cope in an age where they’re being bombarded with too much information
- nickm12 2y ago
  Says someone who has never experienced a data loss bug.
  - silisili 2y ago
    Was the data really important? Is it worth holding people's data hostage because of subpar developers? I'm not at all saying that soft deletion shouldn't exist. I'm arguing that hard deletion should be the default, and then soft deletion practices should have some justification. Instead the industry chose to just soft delete everywhere, which is good for the industry, and bad for users.
    - fc417fc802 2y ago
      Soft delete is a sensible default to prevent all sorts of problems. It's almost always good for a human to be able to get into the system and revert things if something bad happened. It should just be followed up by hard delete several days or weeks later. The general policy should never be to hold things indefinitely. As a practical example I have soft deletion for more or less everything on my desktop. There are periodic filesystem level snapshots and those stick around until I manually GC them. It has saved me more than once.
    - nickm12 1y ago
      The industry chose the default of "data that is stored in durable storage is probably important". Sorry, I just don't see how that is bad for users when compared to the alternative default of "data that is stored in durable storage should be irretrievably deleted immediately".
- groestl 2y ago
  Not deleting something you should have: a fine. In some countries. Deleting something you should not have: potentially a business extinction event. And a fine as well.
- timeon 2y ago
  Seems risky to put your data on non-EU servers. It is still risk, but at least there is legal framework.

carimura 2y ago

it very likely may not be per their own words. They are "bound by various legal and regulatory obligations that may necessitate retention of certain information". This came straight from their privacy team.

Unroasted6154 2y ago

It possible with crypto shredding. You store everything encrypted with one key by customer. When you want to delete you erase the key. The data becomes unusable everywhere (backups included). Then a job periodically garbage collects data without a key, but that's more for cost saving. Big companies do this but it requires some technical maturity. If you operate in Europe you have to implement proper data deletion. I would be more worried about small companies that large ones tombe honest.

Quekid5 2y ago
You still have to backup those keys somewhere... and if you don't do it the same way as for the data then your backups are effectively worthless.
- ripped_britches 2y ago
  Yea was going to say this. I would be surprised if any major company does this for normal uses.
  - Unroasted6154 2y ago
    That's how GAFAMs size companies do it. They need to comply with European regulations for data deletion, and there are very few other options.
- muti 2y ago
  Much less data to back up so it can be stored in a way that is replicated for redundancy but still mutable. Separating the key and data is what allows for sending data to tape backup etc
  - Quekid5 2y ago
    If your (backup-via-redundancy) keys are mutable, you do not have a backup. What happens in the case of a ransomware attack, for example? You've also added (possibly substantial) latency to every single operation that operates on user data.

tdeck 2y ago

Soft deletion doesn't satisfy the company's obligation under the CCPA.