4 ms·
Very sad. But also a miss on admin's part, even if unfortunate. They did not realize during manual restore testing that volume was not being backed up. "Yes an
by bg24 3y ago
Very sad. But also a miss on admin's part, even if unfortunate. They did not realize during manual restore testing that volume was not being backed up.
"Yes and also apparently no. We use #Velero to capture backups of our cluster every 6 hours. From what I had seen our backups had been running successfully. I discovered once the incident started that backups had captured everything but the Persistent Volume Claim data. While manual backup and restore tests were run once a month to ensure our backups were functioning, they were run manually. After digging into why our restores were not coming up with data, I found that our recurring backups were missing the flag to run volume backups with Restic which snapshots PVC block volume data."
- belltaco 3y ago> manual restore testing They didn't test restores of the actual data being backed up. They did a manual restore test of data that they manually backed up.
- appplication 3y agoI want to feel bad for them, and I do, but really I’m not sure what you expect. They never tested their backup. They tested their ability to manually back up, but I’m not sure why they would ever think to conflate that with automatic processes. If they had automatic backups, why were they relying on manual ones for testing?
- gabereiser 3y agoYeah. Kudos to being open about it but this is an amateur mistake. I ran cloud infrastructure for large companies, small companies, and small companies with large data. Disaster Recovery, Disaster Response, Data Backups, Time to restore, etc are all table stakes to running anything in “production”. They tested their ability to do a backup, but never went through with actually backing it up - automating it - let alone making it a process to do before you make changes in “production”. Amateur. I will give this credit though. It’s hard. Not everyone can do it. Running production that is. Anyone can build for production but actually keeping it live, running, and working nominally with proper SLA and DR? It’s hard. I’m curious what the volumes were even for? I tend to not use storage outside of blob, databases, queues.
- bomewish 3y agobut why didn't they just use some normal setup like a postgres database or something? Why all the intense complexity? That makes it less OK
- jrockway 3y agoArgoCD deleting PVCs is pretty much exactly the same as someone clicking around in the web UI deleting EBS volumes (or whatever the equivalent is on their cloud provider). It happens all the time. You have to expect it, or you will lose data. The only thing worse than automation going haywire is someone in a hurry being careless.
- kikimora 3y agoMost of production EBS volumes will be with deletion protection on. Also AWS asks you to explicitly type “delete” when you delete things. I’d blame cloud provider for making it too easy to destroy data and not having built-in backups. But then this is what cheap cloud providers do.
- gabereiser 3y agoVultr has volume protection too but only when in-use. During a deployment you can easily do what the OP did if you don’t do Canary deployments with red/blue failover for zero downtime. The volume will become “non-in-use” and then… The real issue here is it all came down to a tab. “How the backup was created was the key component. Manual backups were made with a command and recurring were setup with a yaml manifest. When we ran manual backups for testing, we set the default-volumes-to-fs-backup flag. The yaml manifest HAD the flag, but it was indented one two many times, so it wasn't actually setting the correct key in the velero helm chart, which might as well have not been setting it at all.”
- diarrhea 3y agoAh. That part was unclear from the text. Thanks for clarifying.