After three years of running it in production I don't really have any horror stories, but we have definitely run into odd and puzzling things. For example, when

After three years of running it in production I don't really have any horror stories, but we have definitely run into odd and puzzling things. For example, when we were experimenting with auto-scaling node-pools on GKE, and the cluster would scale a node-pool up then fail to scale it back down due to the fact that there are daemonsets on every node of a GKE cluster, and this left it in a state where it flopped back and forth between two nodes every ten minutes, evicting the entire workload, trying to delete the node, failing, and resetting. Thing is, we have also run into really bad things in our puppeted infrastructure that we run on vms and manage with terraform. Like the time I accidentally ran a command in the wrong folder and deleted 120 vms :). Well maybe that was just me and a deficiency of our tooling, but anyway one thing is for sure: k8s makes it far easier to recover from situations that arise, once you understand what the problem is and update the cluster with the right state, the controllers will set things right.