3 ms·
It's a very real problem in K8s operators (ArgoCD being a very complicated one, but an operator nonetheless)... ...does absence signify deletion? Some project
by EdwardDiego 3y ago
It's a very real problem in K8s operators (ArgoCD being a very complicated one, but an operator nonetheless)...
...does absence signify deletion?
Some projects decide yes, if the YAML specifying your resource is absent on the next reconciliation, then delete the corresponding K8s resources.
Then some projects go with "no, deletion must be explicit, set this flag in your YAML".
But the last approach doesn't work as nicely with GitOps, as you have to do a two stage workflow to delete - first set the delete flag, then once the resources are deleted, then delete the YAML.
But I'm okay with deletion being harder to do if it makes it harder to well, accidentally delete all your stuff.
Because I've never met a K8s workflow where resources were commonly deleted, usually it's creation and update 90% of the time.
But I can understand it'd be annoying if you ended up with dangling resources if you missed an explicit delete flag.
- ryanjshaw 3y agoDeclarative SQL version control has the same challenge. Best practice now seems to be moving towards explicit SQL migration scripts.
- evanelias 3y agoGuardrails are essential. A properly-designed declarative schema change tool provides guardrails to detect and prevent unwanted deletion / drops / lossy conversions. I develop widely-used software in this space (Skeema) and my tool has always offered drop-prevention functionality ever since its first beta release in 2016. I also argue that guardrails are equally important for imperative migration tools as well, but more often they're lacking or half-baked, which gives a false sense of safety. For example, down/reverse migrations are a very common landmine for human error. Order-of-operations problems also happen frequently when imperative tools are used by large dev teams, resulting in subtle schema drift between environments when there's disagreement between lexicographic migration file order, git history order, and the actual migration application order on each DB.
- ericbarrett 3y agoI think this pattern is fine most of the time. I hate it for PVCs. At the least with cloud-like volumes it should leave a final snapshot that sticks around for a time. I have all the sympathy for the author. I’m sure they don’t feel good right now, but I hope they continue contributing. They’ve learned a hard lesson, but putting it back into their practice is the only way to make it count.
- EdwardDiego 3y agoFully agree. You can use an admission controller to add guard rails, but I'd hope my GitOps operator also offered those guard rails.
- omniglottal 3y agoIt does. The option is called "prune" and absent resources will be deleted ("pruned") when this is set to "true".
- zeeZ 3y agoYou can set a PVC's reclaim policy to "Retain" (default is "Delete"). When the namespaced claim is deleted, the volume will still be around until it is manually deleted. The docs even suggest to do this for "precious data".
- tatersolid 3y agoWhy is catastrophically unsafe behavior the default? This is Ford Pinto-level engineering on the part of the k8s team.
- EdwardDiego 3y agoGuess it's the ol 'cattle not pets' taken to the extreme.