4 ms·

Log files are easy (from a design perspective). Keep identifying information (IP, UID, DoB, etc) in one table, and activity data in another, with a join linking

Log files are easy (from a design perspective). Keep identifying information (IP, UID, DoB, etc) in one table, and activity data in another, with a join linking the two. When user asks to be forgotten, simply overwrite all the identifying information with generated data. Backups work the same way, but depending on where and how those are stored it could get expensive. We're seperating personally indentifying information out of our normal backups (held warm -> cold over time), and keeping PII backups offsite as warm data, so we can delete/ammend relatively pain-free. Yes. GDPR compliance will cost money and I'm not trivialising that. The degree of panic it's causing baffles me though. Sure, it requires design up front. Oh woe. But regulatory compliance always does. Ref. previous rending of hair and gnashing of teeth in the run-ups to Glass–Steagall, HIPAA, Sarbanes–Oxley, and so on.

je42 9y ago
i think we are allow to log the IP etc. as long as there is no link to another identifying key in the same log.
TeMPOraL 9y ago
> The degree of panic it's causing baffles me though. I think it's a combination of two things. On the one hand, people are justifiably worried whether or not their design and implementation of necessary changes will be fully correct. The cost of getting that wrong will be huge, and laws are generally not known for being clean and unambiguous. On the other hand, you also have people panicking because their user-hostile practices are no longer legal, and so they'll lose money.
- Spearchucker 9y ago
  You're right, I think. If anyone worries about whether their design is correct or not it's because they've never treated PII with the respect it deserves. This is doubly so for entities engaging in user-hostile practices. And it's a sad state of affairs. ~12 years ago, in 2005, Kim Cameron first published his 7 laws of identity[1]. Follow those and you'll ace GDPR. Instead we've been directing the sharpest, brightest and smartest minds on the planet at squeezing revenue from advertising, consequences be damned. [1] https://www.identityblog.com/stories/2005/05/13/TheLawsOfIdentity.pdf https://www.identityblog.com/stories/2005/05/13/TheLawsOfIde...
  - Silhouette 9y ago
    If anyone worries about whether their design is correct or not it's because they've never treated PII with the respect it deserves. Maybe that should be true, but it isn't. As a personal example, let's look at one of my own businesses that is not even remotely in the aggressive advertising or other data mining space. The only PII we keep is basic records of how our systems are used, who our customers are, what financial things happened, email correspondence, and so on. Nothing even remotely controversial. Nothing that more or less any online business wouldn't have to keep, often due to legal requirements. And yet in at least one case we're still worried, because having read the actual GDPR, it is far from clear that things like our backup and logging systems aren't technically in violation. We don't know exactly how we would react if a customer did suddenly request deletion of everything we held on them, because we don't know exactly what our legal obligations would be in that situation. Clearly we couldn't just delete everything, because we'd be required to keep a lot of that data for legal and accounting reasons. So if nothing else, we'd then have to spend hundreds or even thousands of pounds on taking expert legal advice, and then potentially hundreds or thousands more to update our systems to remove whatever we had to remove. This is a side business run by a few people, part-time, like many other small or microbusinesses all over Europe. We're not dealing with anything like the scale of personal data of the Facebooks and Googles of the world. Apparently our carefully designed systems, which only collect a reasonable amount of data for legitimate purposes and take care to handle it securely, are already doing significantly better than some of the biggest data hoarding organisations in the world. And yet, the effort and overheads required to address GDPR could pose an existential threat to the business. Tell me how that isn't disproportionate and how it's all clear and been thought through.
    - Spearchucker 9y ago
      Hey I have no idea what your business is, but every case I've come across where people worry about GDPR is clearcut. As you say, you must absolutely hold onto transaction history for tax purposes, at least! That said, unless you're selling arms or something (if compliance with the law is ambiguous go see a lawyer) you must honour an individual's right to be forgotten. You can do this without deleting records of transactions with that individual. You would however note that you can no longer identify the buyer because of such request x on date y. That should be as simple as deleting PII and then ammending the transaciton record with a comment. Your biggest challenge here is how to handle the customer record. Do you delete it outright, and point the buyer field in the transaction record at a single GDPR ghost customer, or do you overwrite the customer data with random, non-PII values (I've always gone with the former)? If your architecture is solid then these change requests should form part of ongoing BAU, no? Submit change request, develop, test, deploy...
yummybear 9y ago
As far as I know (been told by legal), user identifiers are also categorized as personal information.
kalms 9y ago
It sounds like you just made a simple log into something that definitely belongs to the GDPR ruleset! But that's just my interpretation. Can anyone, with citation, please shed some light on this. What's fair use, in broad terms, when it comes to simple HTTP logs or similar structures?
- kuschku 9y ago
  For HTTP logs, allowed use would be e.g. stripping the last octet of an IPv4, or stripping the last 64 to 80 bytes of an IPv6. That’s generally not identifying a single person anymore, and usually good enough for anything else.
  - iovrthoughtthis 9y ago
    This does not make the data not PII as if used in conjunction with some other data it could still be used to identify a natural person.
    - kuschku 9y ago
      If you have such additional data, yes. In the general case (e.g. hosting a simple website where you strip user identifiers and the last octet out of the nginx logs) it becomes no PII then.
  - eadmund 9y ago
    > For HTTP logs, allowed use would be e.g. stripping the last octet of an IPv4, or stripping the last 64 to 80 bytes of an IPv6. > That’s generally not identifying a single person anymore, and usually good enough for anything else. 'Usually'? Even if true (highly doubtful), that's not the same as 'always.' The whole purpose of logs is to be truthful accounts of pertinent data. A full IP address is a pertinent datum. I'm going to step up on my soapbox and assert that any law which forbids me from indelibly recording that 192.0.2.17 requested /all-your-records-are-belong-to-us is a bad law.
    - lokedhs 9y ago
eadmund 9y ago
> Keep identifying information (IP, UID, DoB, etc) in one table, and activity data in another, with a join linking the two. You just made the log no longer simple. Now writing a log line isn't a matter of writing bytes to disk; it's a matter of looking up PII in one table, reading a key, then writing the key and the non-PII to disk. If the definition of PII gets more expansive … whoops. > The degree of panic it's causing baffles me though. Sure, it requires design up front. Oh woe. But regulatory compliance always does. You answered your question. The compliance costs, and it's for what is IMHO a non-right. I do not, and no-one has, a right to demand that my IP activity be erased. I do not, and no-one has, a right to change history. Much of the rest of the GDPR is wonderful stuff, but the so-called 'right to be forgotten' is Orwellian.

je42 9y ago

i think we are allow to log the IP etc. as long as there is no link to another identifying key in the same log.

TeMPOraL 9y ago

> The degree of panic it's causing baffles me though. I think it's a combination of two things. On the one hand, people are justifiably worried whether or not their design and implementation of necessary changes will be fully correct. The cost of getting that wrong will be huge, and laws are generally not known for being clean and unambiguous. On the other hand, you also have people panicking because their user-hostile practices are no longer legal, and so they'll lose money.

Spearchucker 9y ago
You're right, I think. If anyone worries about whether their design is correct or not it's because they've never treated PII with the respect it deserves. This is doubly so for entities engaging in user-hostile practices. And it's a sad state of affairs. ~12 years ago, in 2005, Kim Cameron first published his 7 laws of identity[1]. Follow those and you'll ace GDPR. Instead we've been directing the sharpest, brightest and smartest minds on the planet at squeezing revenue from advertising, consequences be damned. [1] https://www.identityblog.com/stories/2005/05/13/TheLawsOfIdentity.pdf https://www.identityblog.com/stories/2005/05/13/TheLawsOfIde...
- Silhouette 9y ago
  If anyone worries about whether their design is correct or not it's because they've never treated PII with the respect it deserves. Maybe that should be true, but it isn't. As a personal example, let's look at one of my own businesses that is not even remotely in the aggressive advertising or other data mining space. The only PII we keep is basic records of how our systems are used, who our customers are, what financial things happened, email correspondence, and so on. Nothing even remotely controversial. Nothing that more or less any online business wouldn't have to keep, often due to legal requirements. And yet in at least one case we're still worried, because having read the actual GDPR, it is far from clear that things like our backup and logging systems aren't technically in violation. We don't know exactly how we would react if a customer did suddenly request deletion of everything we held on them, because we don't know exactly what our legal obligations would be in that situation. Clearly we couldn't just delete everything, because we'd be required to keep a lot of that data for legal and accounting reasons. So if nothing else, we'd then have to spend hundreds or even thousands of pounds on taking expert legal advice, and then potentially hundreds or thousands more to update our systems to remove whatever we had to remove. This is a side business run by a few people, part-time, like many other small or microbusinesses all over Europe. We're not dealing with anything like the scale of personal data of the Facebooks and Googles of the world. Apparently our carefully designed systems, which only collect a reasonable amount of data for legitimate purposes and take care to handle it securely, are already doing significantly better than some of the biggest data hoarding organisations in the world. And yet, the effort and overheads required to address GDPR could pose an existential threat to the business. Tell me how that isn't disproportionate and how it's all clear and been thought through.
  - Spearchucker 9y ago
    Hey I have no idea what your business is, but every case I've come across where people worry about GDPR is clearcut. As you say, you must absolutely hold onto transaction history for tax purposes, at least! That said, unless you're selling arms or something (if compliance with the law is ambiguous go see a lawyer) you must honour an individual's right to be forgotten. You can do this without deleting records of transactions with that individual. You would however note that you can no longer identify the buyer because of such request x on date y. That should be as simple as deleting PII and then ammending the transaciton record with a comment. Your biggest challenge here is how to handle the customer record. Do you delete it outright, and point the buyer field in the transaction record at a single GDPR ghost customer, or do you overwrite the customer data with random, non-PII values (I've always gone with the former)? If your architecture is solid then these change requests should form part of ongoing BAU, no? Submit change request, develop, test, deploy...

yummybear 9y ago

As far as I know (been told by legal), user identifiers are also categorized as personal information.

kalms 9y ago

It sounds like you just made a simple log into something that definitely belongs to the GDPR ruleset! But that's just my interpretation. Can anyone, with citation, please shed some light on this. What's fair use, in broad terms, when it comes to simple HTTP logs or similar structures?

kuschku 9y ago
For HTTP logs, allowed use would be e.g. stripping the last octet of an IPv4, or stripping the last 64 to 80 bytes of an IPv6. That’s generally not identifying a single person anymore, and usually good enough for anything else.
- iovrthoughtthis 9y ago
  This does not make the data not PII as if used in conjunction with some other data it could still be used to identify a natural person.
  - kuschku 9y ago
    If you have such additional data, yes. In the general case (e.g. hosting a simple website where you strip user identifiers and the last octet out of the nginx logs) it becomes no PII then.
- eadmund 9y ago
  > For HTTP logs, allowed use would be e.g. stripping the last octet of an IPv4, or stripping the last 64 to 80 bytes of an IPv6. > That’s generally not identifying a single person anymore, and usually good enough for anything else. 'Usually'? Even if true (highly doubtful), that's not the same as 'always.' The whole purpose of logs is to be truthful accounts of pertinent data. A full IP address is a pertinent datum. I'm going to step up on my soapbox and assert that any law which forbids me from indelibly recording that 192.0.2.17 requested /all-your-records-are-belong-to-us is a bad law.
  - lokedhs 9y ago

eadmund 9y ago

> Keep identifying information (IP, UID, DoB, etc) in one table, and activity data in another, with a join linking the two. You just made the log no longer simple. Now writing a log line isn't a matter of writing bytes to disk; it's a matter of looking up PII in one table, reading a key, then writing the key and the non-PII to disk. If the definition of PII gets more expansive … whoops. > The degree of panic it's causing baffles me though. Sure, it requires design up front. Oh woe. But regulatory compliance always does. You answered your question. The compliance costs, and it's for what is IMHO a non-right. I do not, and no-one has, a right to demand that my IP activity be erased. I do not, and no-one has, a right to change history. Much of the rest of the GDPR is wonderful stuff, but the so-called 'right to be forgotten' is Orwellian.