9 ms·
Ask HN: How are you implementing GDPR-compliant soft deletes?
The idea: Customer requests account or item deletion, you set it to "deleted" in DB without actually deleting it and it helps for documentation purpose should the dispute arise over some issue in future.
- snowwolf 9y agoSoft deletes as you describe aren't allowed under GDPR. But a possible solution may be to disassociate the data from the customer (as long as the data itself isn't considered 'personal data'). For example, if the reason the data falls under GDPR is because it is connected to the customers email address, you could clear the email address. But that wouldn't let you ever re-associate it. But you could maybe (but don't take my word for it) one way hash the email address (bcrpyt, etc.) and if a dispute arises in the future scan the hashes for a match of the person raising the disputes email address in order to re-associate the data.
- politelemon 9y agoThis is the guidance we have been given as well. It'll depend on the kind of data you're talking about. If you are obligated to keep that data for legal purposes then there's no working around that. But then make an effort to only keep exactly the data you need. However in the context you're asking - absolutely do delete where possible. If not possible then pseudo-anonymization is a way of dealing with this.
- samsari 9y agoMy understanding, as someone implementing the GDPR-compliance for my company right now, is that if you could produce the same one-way hash a second time from the same input email address then the hash is still considered PI.
- tyfon 9y agoYep, we've been through this discussion where I work just a few days ago and one way hashing even with salt is _not_ compliant as you can search for whatever you hashed and get a hit (SSNs, emails etc).
- otp124 9y agoWhile GP stated soft-deletes aren't allowed, I figured I'd contribute to this thought exercise. What about symmetric encrypting the field(s) and then giving the customer the key, and tell them to print it or store it safely, or else they won't be able to recover? And then don't store the key or write it to disk (remove it from memory)
- tyfon 9y agoThen you might as well scramble the data completely as the customer is not supposed to be able to recover the data after deletion.
- mailbag 9y agoWe're going through a similar thought process now for a blockchain based data-store. If the data is symmetrically encrypted and the key is deleted.. does that count as hard deletion? The data is unrecoverable.
- snowwolf 9y agoFair point. I can see the purpose of "right to be forgotten", but I think in some circumstances it is going to be abused. Any service that "bans" users for fraudulent/abusive activity and stores data about the banned user to prevent them from creating new accounts is going to have a problem. Banned user can just request to be forgotten and then create a new account. Unless there is some exception within GDPR that will support this use case.
- deleted 9y ago[deleted]
- mattmanser 9y agoThis is probably covered by article 17, 3, e: https://gdpr-info.eu/art-17-gdpr/ https://gdpr-info.eu/art-17-gdpr/ You're allowed to ignore deletion rules for the purposes of: 3 Paragraphs 1 and 2 [the rights to be forgotten] shall not apply to the extent that processing is necessary: (e) for the establishment, exercise or defence of legal claims.
- pc86 9y agoDo you know how you are supposed to handle disputes in the future? If I ask that all my information be deleted and I say n months later I was charged for something I never received, how does the company disprove that?
- the_mitsuhiko 9y agoYou are legally required to retain payment history anyways for many years. So that's out of the GDPR scope.
- jacquesm 9y agoNo, it is in scope. It is just that the laws are not always consistent. And that's a problem, because you can't be the arbiter of which law takes precedence.
- detaro 9y agoGDPR explicitly lists processing that is necessarily to fulfill legal obligations by european or local law as permitted without further permission and as a reason to deny a deletion request, that covers at least a lot of it. (although you still have to report your use to the user and follow general guidelines on handling sensitive data of course)
- jacquesm 9y agoSo, it is in scope, since it is covered by the law, and then the law explicitly gives you permission to make an exception. But you'd better do all of the following: - have a legally trained person review your eventual solution and your reasoning behind it - document the exceptions, which laws and which datums it covers - keep track of the law as it changes, especially with new bodies of law such as the GDPR you can expect updates to reflect the situation on the ground and in a way the GDPR itself is such a change. - be prepared to review the situation/code if the law changes in the future - be aware that 'data retention' laws are very different from one industry to another (for instance telecommunications is a totally different beast than e-commerce)
- Ezku 9y ago> it helps for documentation purpose should the dispute arise over some issue in future. If you are required to hold on to the data for legal purposes such as dispute settlement, there is no issue. The customer can request you delete such data but you have no obligation to do so. Issues arise when holding on to the data is no longer "necessary". At that point soft deletion is not enough and you must be able to remove personal data regarding that customer from your systems. Source: IP lawyer I talked with last week on this. From what I can tell, one good way of going about this in case you really do not want to throw away data – such as when you're doing event journaling – would be to have all personal data encrypted. When the data reaches its expiration point or is requested for deletion, throw away the decryption key and all you're left with is is the metadata which you are allowed to hold on to for purposes of running your business. EDIT: This concept is called 'cryptographic deletion'. Here's one whitepaper on the subject. http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.397.857 http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.397....
- andriesvh 9y agoAre you sure about throwing away encryption keys is sufficient to be GDPR complient? Does this comes from IP lawyer as well?
- zimpenfish 9y agoI don't think "soft deletes" are actually allowed, are they? https://gdpr-info.eu/art-17-gdpr/ https://gdpr-info.eu/art-17-gdpr/ > the controller shall have the obligation to erase personal data without undue delay
- gtsteve 9y agoAnother pattern you might consider is setting an expiry date on a database row. You only show values where the expiry date is null. Every night you have a cronjob or some other process that deletes all objects that have expired.
- zimpenfish 9y agoNot sure that would comply with the "without undue delay" requirement unless you always set the expiry date to be today and then you might as well just wipe them immediately.
- pan69 9y agoYou probably already add a "time created" column/field to each row/document (at least, I personally do this to all data I store in a database). If you want to expire something, you can now easily calculate it and you can even easily change the timeout/lifespan without having to update the data in the database.
- raverbashing 9y agoHow would that help? Would you set time created to zero?
- pan69 9y agoRather than having an expire date, with a creation time you can e.g. delete everything older than 1 day, or what ever. I.e. you don't hard code the expire time in the data itself. If you want to change the timeout to 2 days it's a change in your delete logic, not your data.
- deleted 9y ago[deleted]
- jjoergensen 9y agoDeletion of backup-data is also an interesting topic
- zimpenfish 9y agoI believe (not a lawyer, etc.) that you don't have to delete from existing backups as long as you have a process to immediately wipe the customer details again if that backup is restored (and before any other processing can happen.) [edit to clarify that you wipe the customer details]
- dijit 9y agoThe way I'm handling it (probably non-compliant) is to store a list of the internal keys we use for people in a list as GDPR requests come in. If I restore a backup it will go via this list and ensure that content in my backups which are keyed to deleted accounts are never restored. In theory we have the data, but it's never reachable by internal systems. -- Anything else is essentially compromising the integrity of a backup.
- jakozaur 9y agoThe law itself was written by someone unaware of that. A lot of interpretations: 1. The most extreme, go back to all of your backups and delete them too. 2. You don't need to do anything, if you do not touch the backups and truly treat them for disaster recovery. 3. Your backups need to have reasonable retention (e.g. two year) and way to apply post requests after recovery. 4. A lot of in between. 5. My personal interpretation is that in first year of GDPR there will be so many companies that are not even trying to be compliant. Any companies showing any reasonable efforts will be just left alone and at worst heard some recommendations. Of course ad-tracking companies might get screwed, but their business model seems to be incompatible with GDPR. Also right to erasure can be tricky (e.g. what if you keep records for support/warranty purpose). What you should do if someone exercise their right to be forgotten and than ask you for refund.
- ZenoArrow 9y agoIn what world is two years worth of backups required for reasonable retention? Either the backups are tiny or the company involved has got more money than sense. I'd see no reason, in any company, for backups to be held for longer than 6 months, and that would be an outside estimate (many companies could get away with only having a couple of months worth of backups).
- chesimov 9y agoWhat would be the sort of dispute you envisage?
- CamperBob2 9y agoGovernment: "You know that data you were required to delete when $(USER) requested to be forgotten? We require you to provide it in connection with our ongoing investigation of $(USER)."
- deleted 9y ago[deleted]
- devdad 9y agoIs this a real issue though? If I comply to regulations to remove data as required by law, I'd be surprised if a government body could require me to provide data I am supposed to have deleted.
- kasey_junk 9y agoThis is a very real GDPR fear. Some of its mandates run counter to other local data retention mandates. It’s not clear yet how that is going to shake out.
- jacquesm 9y agoAs HN'er detaro notes in this comment: https://news.ycombinator.com/item?id=16366864 https://news.ycombinator.com/item?id=16366864 There are some provisions for those situations. And on the subject of backups, those are typically exempt but there are some obvious problems there when you restore a backup at a later time. To me the big ticket items in the GDPR are the notification duty and the data processing agreement 'chain' that gives some level of certainty that the companies you deal with are going to take this serious. The implementation details and all the moving bits and pieces are most likely not going to be the parts where the real tests will be in the first year or two.
- eb0la 9y agoConsult your Data Protection Officer first. GDPR says you must delete information about the customer; but there are cases where you still might need to have that data available. If your customer can interact with another one inside your app/platform, he/she can commit a crime, and you might be required by court (and by law) to disclose some information (even conversations! inside the platform). Setting something to "deleted" might not be the best way to do the actual soft delete. Sometimes you can "delete" that user moving it to a separate part of an LDAP branch (where nobody except someone with authority can access). In other cases, you can add the "deleted" flag on the table. If so, MAKE SURE your app access the data from a view of the table where the 'deleted' users are not present. Even better: partition the underlying table based on the "deleted" field to physically separate active and deleted users. But whatever you do, ask your Data Protection Officer first.
- jacquesm 9y agoI'm going to go out on a limb here and guess that 99% of the companies out there affected by the GDPR and the OP in particular do not have a DPO (yet), and may not realize they need one, and even if they do know that then they likely won't be able to fill the seat either in time or with someone competent. Every year we look at quite a few companies, this is the first year that I've spotted a DPO in the wild, and impressively, they even knew their stuff.
- ust 9y agoNot every company needs a DPO though, e.g. check here: https://www.eugdpr.org/key-changes.html https://www.eugdpr.org/key-changes.html Maybe his company doesn't need one. Of course, whether he has a DPO or not, still the question remains of how to "properly" delete the personal data.
- jacquesm 9y agoIt is quite well possible their company does not need a DPO. But given the nature of the question there is some evidence they do, besides that hiring a DPO is not something done in isolation but most likely as as the result of a GDPR impact study done in ... 2017 or so, which I'm going to again guess was not in the cards for many companies. So, in summary: likely the vast majority of the companies affected is only now starting to wake up to the fact that they are affected, for quite a few of these companies the effects will be relatively benign unless their servers are compromised, for the more serious offender and the larger companies that have not yet started to address these issues it is likely too late to get anything done in time but since this goes for the vast majority of them they are simply playing a complicated game of Russian roulette with the oversight bodies and a couple of them will undoubtedly get lucky to great relief of the remainder. Data protection authorities tend to be vastly understaffed, but this too will hopefully change in the future.
- romanovcode 9y agoYour idea is not allowed under GDPR. You need to unassociate the account from the user. That is: Remove all personal information - email, name, surname, etc..
- rimeice 9y agoIt may be that the future issues you envisage can still be met with anonymous data i.e. some overall audit of your service use. Anonymous data retains a small link to the original but is exempt from GDPR. One method is to generalise records in a database, for example, mask / remove direct identifiers like names, put ages in to age brackets and fuzz spatial data by x distance. So this can be used to deal with erasure subject access requests and sharing data in general. There's a lot of advice on this from the ICO - https://ico.org.uk/media/1061/anonymisation-code.pdf https://ico.org.uk/media/1061/anonymisation-code.pdf Disclaimer: I'm a fan of anonymisation because I'm working on a project to bundle this in to a service - https://anon.ai https://anon.ai - would be great to understand more about your use case.
- no1youknowz 9y agoAbout the GDPR, can anyone recommend a company in the UK they have dealt with, that brought them up to compliance?
- jacquesm 9y agoIf you aim to do this before May 15th you will find that anybody that is capable is fully booked for the remainder of 2018.
- no1youknowz 9y ago> you will find that anybody that is capable Define capable. Look at this thread as an example. Many answers contradict each other. There are so many ways to interpret the guidelines, which in many cases have not been thought through. I have engaged in discussions with 5 companies located in the UK. All gave differing answers on specific questions relating to data for marketing, finance, and fraud.
- jacquesm 9y agoThat was my point. You need to know the law inside out to be able to tell someone exactly what to do in their situation. We - our little band of friends - have been reading up on this subject since the previous privacy law was enacted and all I can tell you is that it is much easier to spot things that are in conflict with the law(s) as written than to come up with a single workable solution that does not leave things open to interpretation. Even so, these laws are good, they will force people to wake up to the underlying issues and to begin to think about their responsibilities when before the mantra seems to have been that any effort spent on privacy and security is better spent on growing the business because otherwise the other guy that doesn't care about those things will eat your lunch.
- no1youknowz 9y ago> That was my point. Was it? Let me be blunt for a second. When someone asks for recommendations. If you are able to help that person, then please do so. It helps them and it may help others in the HN community. What you should not be doing, is your previous reply. In no way was it warranted, nor was it solicited. If you wish to make such statements known. Feel free to post your own comment or replying elsewhere. I cannot believe some elements of HN behave in this manner.
- yread 9y agoWhat if you outsource the PII? For example use a payment processor and only store their reference. You can always go back to transaction in the payment processor in case of disputes, but you don't store the personal information.
- jacquesm 9y agoThen you will need a data processing agreement with that IPSP and they will need to prove to you that they are in compliance. The good news is that in that situation there is probably more knowledge and funding available to get things right (but it is obviously not a guarantee).
- deleted 9y ago[deleted]
- apw808 9y agoLadies, gents... The kinds of things you are saying/asking about GDPR are the same kinds of things being asked on this side of the pond...
- apw808 9y agoOK. Here goes... Advice... Firstly, download the regulation itself. At the very least, read article 17. Article 17 concerns something called "The right to be forgotten". It is about 25 lines of legalese. Once you have read it, have a think about it. Then read it again and have a longer thing about its implications. Then, just to be sure you have not gone stark staring bonkers, read the rest. Carefully. Be under no illusion, the GDPR is a game changer in information management terms, let alone anything else.
- apw808 9y agoBut it is only part of a series of new bits of legislation that is being introduced in Europe related to computer security Some other acronyms "PECR" something all marketeers should have a look at as it governs things like email distribution
- apw808 9y agobecause the GDPR is a tad vague, given the interconnected world we live in nowadays (I am a brit living in a small (very) town called Tidworth which is a pin prick o the map) there is something called "Working Party 29" which is providing some fairly detailed and verbose refinements of definition of key phrases and terms. This is an attorneys bean feast as far as I am concerned because of the vagueness, if nothing else..
- jacquesm 9y agoYou can 'edit' your comments until they are one hour old, so you don't have to make many different top level comments. Also, if you want to make a larger comment it is usually possible to resize the input field.
- raptorcomp 9y agoSome comments on the legal aspect to deletion under the GDPR: 1. deletion can generally only be requested if the personal data is being processed under the individual´s consent. Thus, other personal data such as under legitimate interest or execution of a contract does not fall under it. 2. The rule on deleting the data is not absolute as data retention laws prevail over this rule. Thus, only if no data rentention law mandates the storing of the data (which is often the case for business communication) then you are obliged to delete the data or anonymize it. Hope this clarifies the non-technical aspect. Dominic Staiger https://www.raptorcompliance.com/en https://www.raptorcompliance.com/en
- unkoman 9y agoWe're building a consent framework API so our customers can consent for personal data use. Data is then cleaned and transformed (ETL) from personally identifiable to pseudo-anonymised. The data is also separated into two separate encrypted storages for anonymous and pseudo-anonymised data for generalisation and separation. Random (important) hashed identifiers are created and put into a metadata service which is used as a lookup-table. If right to be forgotten is invoked, the data is disassociated from the pseudo-anonymised and personally identifiable data thus making it anonymous. Important is also how you handle data analytics and this is why we're deploying high restrictions on raw data. Analytics will only be able to be done through an analytics service which can give the employees access to only certain parts of the data which is approved for the use-case. We're using Apache Sentry for fine grained role based authorisation to data and metadata and a directory services for user auth. Things we've learned: * Minimise data usage * Don't use personally identifiable data * You will need to be able to prove consent when it comes to data usage and it cannot be consent by default, it has to be opt-in * Log all data access so that use cases can be proved. This needs to be evaluated and audited * Encrypt in transit and at rest * Centralise mapping for all data
- mr1976 9y agoThis is how it was explained to me, and the approach we take. YMMV. Understand what your lawful basis for storing and processing the data is, and that dictates how you need to handle it. Plenty of people throw around large fines and rights to be forgotten and make sweeping statements about "you can't keep X data/we can hash things!/delete all the things". You have a number of potential reasons for storing or processing data. Legal reasons, contractual reasons, consent based reasons - data subjects have different rights depending on the reason you're storing the data (there is a list of these reasons; see below.) I may be keeping data because I have a contract with a client, and I require the data. An example of this would be an email address stored as a login. My legal basis for storing the information is contractual (I have a contract with my client). Does the data subject (in this case, the user who's email it is) have a right to erasure? No. Can I store the information in my Amazon RDS instance in Virginia? Yes. Provided I've explicitly stated it, and been transparent about how and where I'm storing and processing the data, and my client has agreed. Do I need to secure and look after the data? Yes. Obviously. Do I need to get to get consent from the user? No. That wasn't the legal basis for storing the information. What about consent based stuff? I may have someone subscribe to my mailing list. I get their consent. Therefore, the legal basis for storing and processing is consent. That consent should be time limited, and I should be transparent about it. I need to give them the means to review, withdraw and act on their consent should they wish to. What about keeping a record of a person you've deleted because they have requested it? You can store this. Lawful basis is that it's a legal requirement. If you go and use the data for any other purpose, it's not allowed - because that would require their consent. If you want to understand lawful basis, this is a good overview - https://ico.org.uk/for-organisations/guide-to-the-general-data-protection-regulation-gdpr/lawful-basis-for-processing/ https://ico.org.uk/for-organisations/guide-to-the-general-da... Trying to wade through one half of the GDPR and its requirements without understanding lawful basis leads to confusion, because you'll keep hitting cases that seem completely unreasonable (because they may not be required). Trying to paint the law vague and unreasonable defeats the point - it will become less vague over time. It makes privacy a first class citizen (something we sorely need), and will become more specific as it's tested in the courts, just like any other law in jurisdictions that value legal precedent. Get to know it and work with it - this is not your mother's EU cookie law.
- 9y ago
- he0001 9y agoAll data under GDPR which also falls under another law can be stored, but NOT used for any other purpose than of the overlaying laws. So just because you have the information doesn’t mean you can use it. Having the same information, even “soft deleted” would violate GDPR since you are not allowed to store it like that. Having information grouped like that still implies information which will be under GDPR. Like if you store this information in the “customer” database that will imply that the customer was a customer at some time, and then illegal by GDPR. If you move this information to another database which stores “transactions” which you require to do by law that’s ok. But you are not allowed to query that information to answer the question, “who have been my customers”.
- apw808 9y agoHiya. If I may, a couple more things to add to considerations... Firstly while, rightly, you concentrate on the design impact of article 17. I guess most of you live and work in the US. What that almost certainly means, if you deliver SAAS, is that personal data is exported from the EU for use. You guys really need to understand GDPR Chap 5, articles 44 to 50 inclusive and get hold of something called "treaty 108" which is about the idea of "adequacy". And secondly, I am going to ask someone I know who is a legal eagle to sign up to this site. She is very clever and knows this stuff backwards. She also has an aspiration to come and work in the US....