I've used k8s in startup and now in a large company, so I've seen both worlds. In the cloud, running workloads on k8s is much more expensive than something sim

I've used k8s in startup and now in a large company, so I've seen both worlds. In the cloud, running workloads on k8s is much more expensive than something simpler, like containers on instances in an AutoScaling group on Amazon, for example. You have to pay for the control plane in addition to the worker nodes, and if you only have a few workers, your control plane costs are significant for small infra. It's fairly easy to set up on {A,E,G}KS. This cost, however, buys you flexibility. Say your software needs to run on-prem as well as in the cloud? k8s can be your abstraction, and the additional cost becomes worthwhile. If you use it as a dumb container scheduler, without all this fancy stuff like Persistent Volumes or crazy scheduling constraints, it's not too bad. It does introduce extra complexity around traffic ingress, and is missing some basic functionality, like de-scheduling a Pod from a load balancer, and terminating it once no more connections exist. There are also difficulties around giving people access to the cluster in a safe way (for example, any admin can see the contents of secrets), and kubelets' decision to run without swap also leads to some annoying node failures that would be mere slowdowns if swap was present. Like anything else, k8s is a mixed bag. If it doesn't solve specific problems for you, don't use it, it's pointless.