I have run kafka, postgres, cassandra on K8S but eventually I move off Postgres and Cassandra to normal server. Ideally StatefulSet does help a bit. Such as wi

I have run kafka, postgres, cassandra on K8S but eventually I move off Postgres and Cassandra to normal server. Ideally StatefulSet does help a bit. Such as with StatefulSet you have DNS and hostname like service0.namespace etc. And you can change StatefulManifest without updating the pod. Pod only get updates when we deleted it(OnDelete updating policy) or rolling update with staging parition which mimick real server behaviour where we can pause/restart process on a server. However, what I realized is resource scheduling and EBS volume. 1. Soon I realized the node run db pod should only run db and I make a dedicated node pool for it. When this occurs, It feels like eventually I'm provisioning a server to run this workload. 2. EBS volume cannot mount to other zone. So it really annoying when I kill a pod and it cannot start because the volume cannot attach to node in other zone. And when we need to upgrade Kubernetes itself in an immutable way, mean kill old node and bring up new node. It's a pain to control that process carefully to avoid casscade node re-balancing/replication. More over, the ability to easily goes in server and edit/tweak config is lost with K8S. We have to use ConfigMap, some trick of init container, entry point script to generate custom configuration file etc. An example is broker id in case of kafka or slave-id in case of MySQL. In other words, I feel like running stateful service on K8S is no longer a joy. Once I move these stateful service out of K8S, suddenly everything is so smooth. Running and upgrading K8S itself become a walk in the part. Also, on AWS, when you have a large amount of server, the chance that you got AWS notification about node replacement/rebooting (old hardware, host migration) is very frequent. Dealing with these when all of node have stateful service running on is not easy.