5 ms·
Wow... tooling that deletes a bunch of stuff because a manifest file is missing. Just... wow. I feel for the admin here. This seems equivalent to an accidental
by pdntspa 3y ago
Wow... tooling that deletes a bunch of stuff because a manifest file is missing. Just... wow.
I feel for the admin here. This seems equivalent to an accidental `sudo rm -rf /`.
- kuchenbecker 3y agoYeah, it's a hard lesson. We would always safe-expire or soft-delete to avoid this so each operation is undo-able for ~2 weeks.
- nighmi 3y agoHow do you set this up?
- kuchenbecker 3y agoIn a nutshell, we "abandon" the resource and clean up abandoned resources after N days. To delete something, you need to explicitly check it's name into a list of resources to delete. So deletions are explicit and a mistake is just un-abandoning.
- defaultcompany 3y agoUgh I did something like this once when I wrote a script to rsync one volume to a backup volume. Everything was great until someone accidentally rm -r ‘d the source volume and the script happily deleted everything on the backup volume to keep it in sync.
- ryanjshaw 3y agoThat was never a backup volume then, it was a mirrored volume. (Ask me how I learned the distinction.)
- philipswood 3y agoHow did you learn the distinction?
- rokkitmensch 3y agoThe same way we learn the distinction every night, Pinky. By pissing on the electric fence ourselves!
- tedunangst 3y agoxkcd: automate, from auto meaning self, and mate meaning...
- jamiepenney 3y agoMate means death in te reo Māori, which also kinda works here.
- lokar 3y agoYeah, it should not delete everything in that situation, it should require a tombstone file.
- EdwardDiego 3y agoIt's a very real problem in K8s operators (ArgoCD being a very complicated one, but an operator nonetheless)... ...does absence signify deletion? Some projects decide yes, if the YAML specifying your resource is absent on the next reconciliation, then delete the corresponding K8s resources. Then some projects go with "no, deletion must be explicit, set this flag in your YAML". But the last approach doesn't work as nicely with GitOps, as you have to do a two stage workflow to delete - first set the delete flag, then once the resources are deleted, then delete the YAML. But I'm okay with deletion being harder to do if it makes it harder to well, accidentally delete all your stuff. Because I've never met a K8s workflow where resources were commonly deleted, usually it's creation and update 90% of the time. But I can understand it'd be annoying if you ended up with dangling resources if you missed an explicit delete flag.
- ryanjshaw 3y agoDeclarative SQL version control has the same challenge. Best practice now seems to be moving towards explicit SQL migration scripts.
- evanelias 3y agoGuardrails are essential. A properly-designed declarative schema change tool provides guardrails to detect and prevent unwanted deletion / drops / lossy conversions. I develop widely-used software in this space (Skeema) and my tool has always offered drop-prevention functionality ever since its first beta release in 2016. I also argue that guardrails are equally important for imperative migration tools as well, but more often they're lacking or half-baked, which gives a false sense of safety. For example, down/reverse migrations are a very common landmine for human error. Order-of-operations problems also happen frequently when imperative tools are used by large dev teams, resulting in subtle schema drift between environments when there's disagreement between lexicographic migration file order, git history order, and the actual migration application order on each DB.
- ericbarrett 3y agoI think this pattern is fine most of the time. I hate it for PVCs. At the least with cloud-like volumes it should leave a final snapshot that sticks around for a time. I have all the sympathy for the author. I’m sure they don’t feel good right now, but I hope they continue contributing. They’ve learned a hard lesson, but putting it back into their practice is the only way to make it count.
- skoob 3y agoBy default, ArgoCD doesn't remove resources if the manifests are deleted, but you can switch this by turning on auto prune[1]. Likewise, you can specify whether a persistent volume should be retained or deleted if the k8s PersistentVolume resource is deleted[2]. Both of those settings would have prevented data loss in this case, but of course that's easy to say in hindsight. [1] https://argo-cd.readthedocs.io/en/stable/user-guide/auto_sync/#automatic-pruning https://argo-cd.readthedocs.io/en/stable/user-guide/auto_syn... [2] https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaiming https://kubernetes.io/docs/concepts/storage/persistent-volum...
- mannyv 3y agoTerraform will do this. It's just like a command line, but for your environment.
- tetha 3y agoTerraform, unless invoked as tf-yolo (aka terraform -auto-approve), will present you with a cheerful "Hello, I would like to delete everything. Continue: y/n". This makes it fairly easy to panic before deleting everything. Otoh, if you run <terraform destroy> and don't read the plan carefully... eh. We're planning to streamline our run approvals a bit, because approving every DNS record addition results in some approval fatigue, but resource deletion will most likely always require human approval.
- mannyv 3y agoTrue. I carefully look over tf apply to ensure it's doing what i want. But other people are problematic.
- hotsauceror 3y agoAgreed. Reading this kind of thing makes my blood run cold. Your 'accidental sudo rm -rf /' not only destroyed one machine, but all of them. The entire cluster, standbys, backups, storage volumes, the whole enchilada. That's a Business Ending Event, in one merge request. I'm hard-pressed to think of a scenario where a single sysadmin in an 'old-fashioned' enterprise datacenter could do as much damage with, maybe not exactly a one-liner but a lightweight change instruction. The risk-reward calculation seems completely bananas to me.