6 ms·

The data protection regulations really are so freeing, huh. It's amazing to be able to delete all this stuff without worrying about having to keep it forever.

whimsicalism 5y ago
now this is a spin i havent heard before.
- hvs 5y ago
  You haven't heard it because it's not spin, it's from an engineer's point of view. That's not the view you hear in the news when it comes to these things.
  - alisonkisk 5y ago
    Eh, Retention and Deletion are both pain for devs. Not having to care is the happy state.
  - whimsicalism 5y ago
    HN seems like an odd place to assume that people only hear about things from the news and aren't engineers themselves. i am a dev that has to deal with these regulations in my day to day. it is a pain, it is not freeing in any sense, and it makes my models worse. granted, i think there are good reasons for it, but it does not make my life easier for sure.
- jabroni_salad 5y ago
  As a sysadmin I really wish you had. SO MANY problems have come to my desk because some dude 3 years ago did not consider retention or rotation and now I have to figure out what to do with a 4TB .txt that is apparently important.
  - briffle 5y ago
    "You never know when you might need this info to debug" The developer says as their cronjob creates a 250MB csv file, and a few MB of debug logs per day, for the past few years. "Disk is cheap" they say. As a sysadmin, I hate that too.
    - whimsicalism 5y ago
      sometimes the data is just big...
      - colechristensen 5y ago
        Often a considerable portion of those logs are useless, trace level misclassified as info, kept for years for no reason. You should keep a minimal set of logs necessary for audit, logs for errors which are actually errors, and logs for things which happen unexpectedly. What people do keep are logs for everything which happens, almost all of which is never a surprise. One needs to go through logs periodically and purge the logging code for every kind of message which doesn’t spark joy, I mean seem like it would ever be useful to know.
        whimsicalism 5y ago
        sure, in a world where machine learning doesnt exist i would agree with you. for low level logs of things like "memory low, spawning a new container" i would also agree with you. not for user actions though (which is the topic closest to whats under discussion given what sort of data these regulations cover)
  - dylan604 5y ago
    Find out how important it is with a `mv 4TB.txt 4TB.old` type of things. See how many people come screaming
  - chrisjc 5y ago
    Have you come up with a process, or an idea for a process to ensure this doesn't happen? For instance when they create a provisioning request, are you able to set an extremely low threshold? When they say that won't do, the cost increases and their able to see/understand and start to care about the actual lifecycles of what they're creating? Surely there is a way to project and monitor the cost of their resources over time, and deliver them an invoice on a regular basis? In other words something like a cost attribution model? That way when the bills start to increase dramatically overtime, pinpointing the heavy hitters becomes trivial, and when they come knocking on your door to "do something about it" you can just say "go talk to Bob". I don't mean to sound like I'm trivializing the problem (honestly I can relate as I've gone through it myself), but I'd love to hear how anyone else has dealt with this issue effectively.
jeff_vader 5y ago
In case of my previous employer it led to incredibly complicated encryption system. It took couple years to maybe implement in 10% of the system. Deleting any old data was rejected.
- hinkley 5y ago
  I wonder sometimes if it would help if we collectively watched more anti-hoarding shows, in order to see how the consultants convince their customers they can get rid of stuff.
  - mro_name 5y ago
    humans started their first 300k years as nomads – storing was just impossible and decrufing happened by itself when moving along. So maybe that's why we're not good at it yet.
    - hinkley 5y ago
      Being a renter definitely kept me lighter for a long time. When you have to box things up over and over you find that the physical and mental energy around keeping it aren’t adding up. I wonder if migrating from cloud to cloud would simulate this experience.
      - travisgriggs 5y ago
        Two space garbage collector in action right there. Maybe all things software need a "move it or lose it" impetus. Features in apps, old data, you name it. If you've gotta keep transferring/translating it, it would definitely pare things down.
        mro_name 5y ago
        maybe reuse is inferior to re-implement. Moderately re-inventing wheels may be beneficial. What may be a threshold?
      - 5y ago
theshrike79 5y ago
Yep, having everything disappear at 2 months max is a life-saver. That "absolutely essential thing" isn't essential any more when there is a possible GDPR/CCPA violation with a significant fine just around the corner.
- koolba 5y ago
  Just make sure you actually test your backups. Two months of unusable backups are just as useful as no backups.
  - marcosdumay 5y ago
    Well, you should have done this before GDPR too, but reminding people to test backups is never too late and never too often.

whimsicalism 5y ago

now this is a spin i havent heard before.

hvs 5y ago
You haven't heard it because it's not spin, it's from an engineer's point of view. That's not the view you hear in the news when it comes to these things.
- alisonkisk 5y ago
  Eh, Retention and Deletion are both pain for devs. Not having to care is the happy state.
- whimsicalism 5y ago
  HN seems like an odd place to assume that people only hear about things from the news and aren't engineers themselves. i am a dev that has to deal with these regulations in my day to day. it is a pain, it is not freeing in any sense, and it makes my models worse. granted, i think there are good reasons for it, but it does not make my life easier for sure.
jabroni_salad 5y ago
As a sysadmin I really wish you had. SO MANY problems have come to my desk because some dude 3 years ago did not consider retention or rotation and now I have to figure out what to do with a 4TB .txt that is apparently important.
- briffle 5y ago
  "You never know when you might need this info to debug" The developer says as their cronjob creates a 250MB csv file, and a few MB of debug logs per day, for the past few years. "Disk is cheap" they say. As a sysadmin, I hate that too.
  - whimsicalism 5y ago
    sometimes the data is just big...
    - colechristensen 5y ago
      Often a considerable portion of those logs are useless, trace level misclassified as info, kept for years for no reason. You should keep a minimal set of logs necessary for audit, logs for errors which are actually errors, and logs for things which happen unexpectedly. What people do keep are logs for everything which happens, almost all of which is never a surprise. One needs to go through logs periodically and purge the logging code for every kind of message which doesn’t spark joy, I mean seem like it would ever be useful to know.
      - whimsicalism 5y ago
        sure, in a world where machine learning doesnt exist i would agree with you. for low level logs of things like "memory low, spawning a new container" i would also agree with you. not for user actions though (which is the topic closest to whats under discussion given what sort of data these regulations cover)
- dylan604 5y ago
  Find out how important it is with a `mv 4TB.txt 4TB.old` type of things. See how many people come screaming
- chrisjc 5y ago
  Have you come up with a process, or an idea for a process to ensure this doesn't happen? For instance when they create a provisioning request, are you able to set an extremely low threshold? When they say that won't do, the cost increases and their able to see/understand and start to care about the actual lifecycles of what they're creating? Surely there is a way to project and monitor the cost of their resources over time, and deliver them an invoice on a regular basis? In other words something like a cost attribution model? That way when the bills start to increase dramatically overtime, pinpointing the heavy hitters becomes trivial, and when they come knocking on your door to "do something about it" you can just say "go talk to Bob". I don't mean to sound like I'm trivializing the problem (honestly I can relate as I've gone through it myself), but I'd love to hear how anyone else has dealt with this issue effectively.

jeff_vader 5y ago

In case of my previous employer it led to incredibly complicated encryption system. It took couple years to maybe implement in 10% of the system. Deleting any old data was rejected.

hinkley 5y ago
I wonder sometimes if it would help if we collectively watched more anti-hoarding shows, in order to see how the consultants convince their customers they can get rid of stuff.
- mro_name 5y ago
  humans started their first 300k years as nomads – storing was just impossible and decrufing happened by itself when moving along. So maybe that's why we're not good at it yet.
  - hinkley 5y ago
    Being a renter definitely kept me lighter for a long time. When you have to box things up over and over you find that the physical and mental energy around keeping it aren’t adding up. I wonder if migrating from cloud to cloud would simulate this experience.
    - travisgriggs 5y ago
      Two space garbage collector in action right there. Maybe all things software need a "move it or lose it" impetus. Features in apps, old data, you name it. If you've gotta keep transferring/translating it, it would definitely pare things down.
      - mro_name 5y ago
        maybe reuse is inferior to re-implement. Moderately re-inventing wheels may be beneficial. What may be a threshold?
    - 5y ago

theshrike79 5y ago

Yep, having everything disappear at 2 months max is a life-saver. That "absolutely essential thing" isn't essential any more when there is a possible GDPR/CCPA violation with a significant fine just around the corner.

koolba 5y ago
Just make sure you actually test your backups. Two months of unusable backups are just as useful as no backups.
- marcosdumay 5y ago
  Well, you should have done this before GDPR too, but reminding people to test backups is never too late and never too often.