6 ms·

Whenever someone publishes an article like this, I want them to find out what its like when you reach the threshold of 100000 individual delete requests per sec

• jacquesm 6y ago
  I find it very interesting that given a chance to hover up a billion records we see no technical challenge at all but when asked to get rid of them suddenly you're demanding unobtanium. If you can't reliably get rid of the data you administer, don't collect it in the first place.
• justwalt 6y ago
  I’m hesitant to think they just couldn’t figure out how to make it work. Giving that company the benefit if the doubt seems like a bad idea.
• GoodJokes 6y ago
  Whenever someone writes a comment like this complaining that something is too hard I always want to remind them that that isn’t an excuse.
• kuschku 6y ago
  They've got 30 days to delete data, there's no excuse not to manage it within 30 days. Also, you're saying they intentionally built a software stack without even thinking about how to obey the law (the 30-day deletion requirement existed even before the GDPR, since 1996 even, just with lower fines) I hope in a few years this case will be taught in school just like the Therac case is being taught right now.
  • holidayacct 6y ago
    It's not that simple, some of these companies have tech stacks that are run-a-way trains written by people who had good intentions but wrote their code and built their infrastructure in a way that was meant for 1/50th of the traffic they currently have. I don't know anything about instagram's infrastructure but what I can tell you for certain is that every one of these companies has legacy infrastructure and code that keeps them from being able to meet all legal requirements. You can't just make a blanket statement that they don't have an excuse.
    • s1t5 6y ago
      > I don't know anything about instagram's infrastructure but what I can tell you for certain is that every one of these companies has legacy infrastructure So you don't know anything about Instagram but at the same time know for certain that they have legacy infrastructure that keeps them from meeting legal requirements? How about we don't give them a pass just because it requires a bit of work to comply with laws and regulations.
      • holidayacct 6y ago
        Maybe its just me but I have yet to work for a company that didnt have legacy infrastructure that was hold them back in some way. Even the startups I've worked for had to make odd business decisions in order to take on new customers. Those odd one-offs and business decisions eventually led to sprawling legacy infrastructure. If there is an infrastructure Utopia I have yet to work there or hear of it.
    • ethanwillis 6y ago
      If they don't want to comply with laws then they shouldn't be operating.
    • LunaSea 6y ago
      That really isn't the user's problem. Laws and rules need to be followed no matter what. It's the same with GDPR.
• stevievee 6y ago
  This rationalization isn't acceptable to me. They have gladly set up the infrastructure to handle a high volume of new information and it should go both ways.
  • anon73044 6y ago
    should See RFC 2119, "should" usually denotes something as "optional" But really, almost zero companies build software around data in such a way that the data could and will disappear at any moment.
    • paulryanrogers 6y ago
      Their comment wasn't an RFC. Right to be forgotten and GDPR are not suggestions. Regardless, in practice, billion dollar companies have the resources to ensure 'delete' isn't actually "partially hide for years/forever". With proper surrogate IDs they could even preserve a minimal amount of meta such as tombstone IDs to ensure all first and third party systems are in compliance.
• loriverkutya 6y ago
  Whenever I see someone write a comment like this, I want him to understand, that don't say deleted, if it's not deleted. Also, if it not works at scale, don't tell your users that it works and pretend it works as you communicated. I find really bothering, when engineers trying to hide behind the "it's a very complicated process in the background, so we just say we have done something, when the truth is, it is in progress. It's not a user error, when you see the word "deleted" you assume, your picture is gone, it's your error, if you state it's gone, while it is not.
  • foepys 6y ago
    Developers are also falling into this trap regularly with distributed systems. MongoDB famously claimed to be "eventually consistent" but in reality failed to deliver on this promise. The MongoDB authors then tried to weasel their way out of it the same way you are describing and many developers rightfully called them out on it.
  • holidayacct 6y ago
    I agree with you completely, no one should be hiding behind "It's a really complicated process". They need to actually sit down and come up with a concrete plan on how to tackle the problem then communicate that we have a plan in place its been added to the roadmap and we are going to tackle it in Q3 20XX.
  • notyourwork 6y ago
    Bingo! Its the same as saying we take your username and password and let you login. Do you or do you not? Its a boolean feature that either does or doesn't do what you say it does. Deletion can be complicated and if it is provide that indication to your users. Software is about setting the correct expectations, otherwise we leave users to interpret written language which we all know leaves room for misinterpretation due to all sorts of complexities with the way humans communicate with one another.
• s_dev 6y ago
  > you literally can't just delete data You literally can. You can't remove data that you don't control though e.g. Removing embarrassing pictures from the internet. However if you control the server you control the data. I can appreciate that it might not be as simple as a unix command but it is a single letter in CRUD. It's fundamental.
• ctvo 6y ago
  Hi. I also work at this scale. It doesn't take a year to delete the data. Be upfront with your users: 'Your delete request is being processed. No one will be able to see this item though it may take up to three days for the data to be completely removed from our servers' This is not a technical challenge, it's an issue of priorities and Facebook doesn't prioritize privacy. They were able to release a TikTok clone in a few months, but can't solve deleting data?
  • throw_me_2020 6y ago
    Deletion is very much a technical/organizational challenge. Imagine some engineer writing some feature you've never heard of and they decide they need data in a different format. They write a job to do this. Now imagine that job somehow has a bug which prevents the deletion system from understanding that copy exists. That bug could be non-obvious to find.
  • parliament32 6y ago
    >It doesn't take a year to delete the data It does if you have cold backups that take a year to cycle out. They're often offsite, compressed, and incrementally hashed so finding individual items and removing them is really hard -- you're much better off just waiting for them to expire, and a year isn't an unreasonable amount of time.
    • slg 6y ago
      The article specifically states this data was downloaded using the "Download Your Information tool on Instagram". The undeleted data wasn't from a cold backup or any backup, it was still on production systems.
      • parliament32 6y ago
        If this is a GDPR "right to access" request, which most "download your information" tools are, then backups are included. See https://law.stackexchange.com/questions/27625/gdpr-compliance-does-it-extend-to-database-backups-and-archived-records https://law.stackexchange.com/questions/27625/gdpr-complianc...
• endtime 6y ago
  Google has an internal standard for how long it takes to wipe out user data after a deletion request, and compliance with it is taken very seriously. It's a lot less than 1 year.
  • jeffbee 6y ago
    I think this points to (one of) the significant flaws in the cult of the "minimal viable product". By eliding up-front design tasks, you may implement something in which complying with the law is impossible. Data retention compliance has to be designed in from the beginning, or it might be technically infeasible to add it later.
    • nthj 6y ago
      I think this is why you need a very senior engineer overseeing your MVP. Before MVPs were championed, companies would create a full-fledged project management suite with 50 database models and release it to crickets. An MBA and a junior engineer define an MVP as “You can log in, and add todo lists and check them off.” A senior engineer defines that same MVP as “You can log in, add todo lists, check them off, delete your account, and if the server crashes or we’re hacked, we have backups for 30 days.” Or in other words with an MVP you still need to complete the 80% of the iceberg the MBA and junior engineer can’t see, but you don’t build the whole iceberg at once.
  • notacoward 6y ago
    ...and I do not believe for one second that they have never discovered a copy of data lying around somewhere after they thought it was deleted. The choice is to admit it, or lie about it.
    • jacquesm 6y ago
      It has actually happened and they did come clean about it. Some data was discovered in log files that shouldn't have been there. They owed up and fixed it pronto even if the chances that the data was leaked outside of Google were nil. There is plenty wrong with Google, but that isn't one of those.
• dhdhhdd 6y ago
  If you can insert... for sure you can delete? It doesn't take days for your picture to show up. Sure, eventual consistency and replication and distributed storage is hard. Alternative approach: encrypt at rest, delete encryption key. It would require extra resources for sure, but if privacy was a concern... ;-) (and then have a reasonable expiry on your CDN if you use one). (of course then we can argue that deleting the encryption key may be difficult too, for the same reasons)
  • jandrewrogers 6y ago
    The cost of insert and delete is massively asymmetrical. This is an intentional and fundamental architectural decision in all of our data infrastructure and built into the data structures and algorithms that are used. Not only does “delete optimized” data infrastructure not exist, in many cases we don’t have good computer science for how you would even design such a thing while having good performance for all the other operations. People that suggest using encryption and throwing away the key have not thought through the implications of that approach. It scales very poorly and therefore is not suitable for most practical systems. There are good reasons “obvious” solutions like this are not used.
    • mtnGoat 6y ago
      I would say a system that can't effectively delete also scales poorly. No?
      • jandrewrogers 6y ago
        It depends on the definition of “delete”. Removing a record from a data model scales just fine, and is how it has always been defined in database systems. Making that record physically unrecoverable from any hardware in the broader system is extraordinarily expensive for fundamental technical reasons. Traditionally “delete” has always meant the former in all systems, and physical deletion is deferred to a point in the future when either the an opportunity arises to do it inexpensively or the cost becomes acceptable. There is a value in recovering the resources consumed by the deleted record but it comes at a very high cost. If you change the definition of “delete” to mean unrecoverable physical deletion, then sure, it scales poorly. But it is a bit like redefining “fast car” to “can travel faster than Mach 5” — technically a valid as a redefinition while completely ignoring the engineering realities of what a car can do.
• parliament32 6y ago
  This. The amount of time quoted in the article in excessive but it's also sparse on details (maybe his data request also sent him backups related to him? a year is reasonable in this case). Deletion is pretty much the hardest thing to do in data management, way harder than inserting or retrieving data. I know GDPR says you need to be able to delete a user's data, but how reasonable is it to trawl through your offsite backups to find their items and remove them?
• soamv 6y ago
  I don't think people understand the vastness of resources that FB and Instagram have at their disposal. This is not a particularly hard problem. Yes it's not a simple SQL query. But they do solve it -- in fact this was a bug that they have already fixed.
• deleted 6y ago
  [deleted]
• Traster 6y ago
  Woe is me. My poor little (looks at notes) trillon dollar company can't cope! Instragram seems to be able to cope with 100000 new photos per second - that they can do. Across geographies and data centres. Ever seen a user facing error due to a newly uploaded photo? No one expects you to just rm -rf. They just expect you to expect the same effort to remove the file that they put in to create the file. It's not like the file got propogated to 10 different data centres by accident. There was a decision that an uploda needed to be everywhere in 5 seocnds and a deletion was an advisory hint that could be ignored for a year.
• gjs278 6y ago
  you’re just a bad programmer if you can’t do this. I could work for your company for a day and solve this
• tgsovlerkhgsel 6y ago
  Just because it's hard doesn't make it optional. If you operate at the scale of Instagram, yes, it becomes very hard, but the companies typically have both the revenues and expert staffing to do it. And if they don't, they're mismanaged and need to allocate more resources to it, because it's illegal not to, for good reason.