My philosophy here is never automate/gitops something that you cannot revert manually if something goes wrong. I learned this almost the same way as OP, but in

My philosophy here is never automate/gitops something that you cannot revert manually if something goes wrong. I learned this almost the same way as OP, but in test and dev environments. As my K8s clusters crashed irrecoverably in many different ways (etcd, pvcs, velero, hdd failures, shared storage, databases) to the point where they'd need to be rebuilt, I imagined what it would be like in prod, and the thought made me quickly decide to: - Not host databases inside k8s - Not host shared storage or storage clusters in k8s - Drop etcd, use external postgres as control plane database - Use velero, but make sure you do no need to rely on velero (backup manifests, not databases or volumes) - Also keep your manifests in git - Generaly treat your entire cluster as ephemeral and optimize for full rebuilds (drive your time to rebuild down to a few hours or minutes).