7 ms·
sigh. My team is facing all these issues. Drowning in data. Crazy S3 bill spikes. And not just S3 - Azure, GCP, Alibaba, etc since we are a multi-cloud product.
by lenkite 5y ago
sigh. My team is facing all these issues. Drowning in data. Crazy S3 bill spikes. And not just S3 - Azure, GCP, Alibaba, etc since we are a multi-cloud product.
Earlier, we couldn't even figure out lifecycle policies to expire objects since naturally every PM had a different opinion on the data lifecycle. So it was old-fashioned cleanup jobs that were scheduled and triggered when a byzantine set of conditions were met. Sometimes they were never met - cue bill spike.
Thankfully, all the new data privacy & protection regulations are a life-saver. Now, we can blindly delete all associated data when a customer off-boards or trial expires or when data is no longer used for original purpose. Just tell the intransigent PM's that we are strictly following govt regulations.
- CydeWeys 5y agoThe data protection regulations really are so freeing, huh. It's amazing to be able to delete all this stuff without worrying about having to keep it forever.
- whimsicalism 5y agonow this is a spin i havent heard before.
- hvs 5y agoYou haven't heard it because it's not spin, it's from an engineer's point of view. That's not the view you hear in the news when it comes to these things.
- alisonkisk 5y agoEh, Retention and Deletion are both pain for devs. Not having to care is the happy state.
- whimsicalism 5y agoHN seems like an odd place to assume that people only hear about things from the news and aren't engineers themselves. i am a dev that has to deal with these regulations in my day to day. it is a pain, it is not freeing in any sense, and it makes my models worse. granted, i think there are good reasons for it, but it does not make my life easier for sure.
- jabroni_salad 5y agoAs a sysadmin I really wish you had. SO MANY problems have come to my desk because some dude 3 years ago did not consider retention or rotation and now I have to figure out what to do with a 4TB .txt that is apparently important.
- briffle 5y ago"You never know when you might need this info to debug" The developer says as their cronjob creates a 250MB csv file, and a few MB of debug logs per day, for the past few years. "Disk is cheap" they say. As a sysadmin, I hate that too.
- whimsicalism 5y agosometimes the data is just big...
- colechristensen 5y agoOften a considerable portion of those logs are useless, trace level misclassified as info, kept for years for no reason. You should keep a minimal set of logs necessary for audit, logs for errors which are actually errors, and logs for things which happen unexpectedly. What people do keep are logs for everything which happens, almost all of which is never a surprise. One needs to go through logs periodically and purge the logging code for every kind of message which doesn’t spark joy, I mean seem like it would ever be useful to know.
- whimsicalism 5y agosure, in a world where machine learning doesnt exist i would agree with you. for low level logs of things like "memory low, spawning a new container" i would also agree with you. not for user actions though (which is the topic closest to whats under discussion given what sort of data these regulations cover)
- dylan604 5y ago
- jeff_vader 5y agoIn case of my previous employer it led to incredibly complicated encryption system. It took couple years to maybe implement in 10% of the system. Deleting any old data was rejected.
- hinkley 5y agoI wonder sometimes if it would help if we collectively watched more anti-hoarding shows, in order to see how the consultants convince their customers they can get rid of stuff.
- mro_name 5y agohumans started their first 300k years as nomads – storing was just impossible and decrufing happened by itself when moving along. So maybe that's why we're not good at it yet.
- hinkley 5y agoBeing a renter definitely kept me lighter for a long time. When you have to box things up over and over you find that the physical and mental energy around keeping it aren’t adding up. I wonder if migrating from cloud to cloud would simulate this experience.
- travisgriggs 5y agoTwo space garbage collector in action right there. Maybe all things software need a "move it or lose it" impetus. Features in apps, old data, you name it. If you've gotta keep transferring/translating it, it would definitely pare things down.
- mro_name 5y agomaybe reuse is inferior to re-implement. Moderately re-inventing wheels may be beneficial. What may be a threshold?
- 5y ago
- theshrike79 5y agoYep, having everything disappear at 2 months max is a life-saver. That "absolutely essential thing" isn't essential any more when there is a possible GDPR/CCPA violation with a significant fine just around the corner.
- koolba 5y agoJust make sure you actually test your backups. Two months of unusable backups are just as useful as no backups.
- marcosdumay 5y agoWell, you should have done this before GDPR too, but reminding people to test backups is never too late and never too often.
- StratusBen 5y agoDisclosure: I'm Co-Founder and CEO of a cloud cost company named https://www.vantage.sh/ https://www.vantage.sh/ - I also used to be on the product management team at AWS and DigitalOcean. I'm not intentionally trying to shill but this is exactly why people choose to use Vantage. We give them a set of features for automating and understanding what they can do to manage and save on costs. We're also adding multi-cloud support (GCP is in early access, Azure is coming) to be a single pane of glass into cloud costs. If anyone needs help on this stuff, I really love it. We have a generous free tier and free trial. We also have a Slack community of ~400 people nerding out on cloud costs.
- samlambert 5y agoVantage is a seriously awesome product. We love it at PlanetScale. Obviously being a cloud product things can get pricy and so Vantage is essential.
- vdm 5y agohttps://www.google.com/search?q=site%3Ahttps%3A%2F%2Fdocs.vantage.sh%2F+s3 https://www.google.com/search?q=site%3Ahttps%3A%2F%2Fdocs.va... I gave vantage.sh 5 minutes and did not see anything for S3 that is not already available from the built-in Cost Explorer, Storage Lens, Cost and Usage Reports, and taking 1 hour to study the docs https://docs.aws.amazon.com/AmazonS3/latest/userguide/BucketBilling.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/Bucket... Most "cloud optimisation" products want to tell you which EC2 instance type to use, but can't actually give actionable advice for S3. Happy to be corrected on this.
- simonw 5y agoSaving people from learning how to use Cost Explorer, Storage Lens, Cost and Usage Reports - and then taking 1 hour to study documentation - sounds to me like a legitimate market opportunity.
- alar44 5y agoNot really. Sometimes you actually have to understand things. If you're so concerned about your billing, someone on your team should probably invest a freaking hour to understand it. If that can't happen, you are just setting yourself up for failure.
- candiddevmike 5y agoAre you multi-cloud because your customers need you to be multi-cloud?
- lenkite 5y agoYes, geographically diverse customers who prefer different cloud platforms.
- raxxorrax 5y agoI host stuff on AWS, but I am pretty sure that hosting on my own server or a server a IT service provider maintains is much cheaper.
- tekknik 5y agoDid you include maintenance, patching and machine upgrades? Cause likely it’s not.