4 ms·

but why didn't they just use some normal setup like a postgres database or something? Why all the intense complexity? That makes it less OK

• jrockway 3y ago
  ArgoCD deleting PVCs is pretty much exactly the same as someone clicking around in the web UI deleting EBS volumes (or whatever the equivalent is on their cloud provider). It happens all the time. You have to expect it, or you will lose data. The only thing worse than automation going haywire is someone in a hurry being careless.
  • kikimora 3y ago
    Most of production EBS volumes will be with deletion protection on. Also AWS asks you to explicitly type “delete” when you delete things. I’d blame cloud provider for making it too easy to destroy data and not having built-in backups. But then this is what cheap cloud providers do.
    • gabereiser 3y ago
      Vultr has volume protection too but only when in-use. During a deployment you can easily do what the OP did if you don’t do Canary deployments with red/blue failover for zero downtime. The volume will become “non-in-use” and then… The real issue here is it all came down to a tab. “How the backup was created was the key component. Manual backups were made with a command and recurring were setup with a yaml manifest. When we ran manual backups for testing, we set the default-volumes-to-fs-backup flag. The yaml manifest HAD the flag, but it was indented one two many times, so it wasn't actually setting the correct key in the velero helm chart, which might as well have not been setting it at all.”