5 ms·

A database is not a service well suited for ephemeral, quick scale down and scale up container workloads that K8s really excells at, and if you have nodes speci

A database is not a service well suited for ephemeral, quick scale down and scale up container workloads that K8s really excells at, and if you have nodes specifically configured to run databases and nothing else then you should simply run them there directly and not mangle K8s to replicate what will essentially be equivalent to running them there directly.

manigandham 5y ago
K8S runs workloads across servers in a declarative fashion. Ephemeral or stateful doesn't make a difference anymore. Whatever you use to manage the nodes directly is basically what K8S provides (and more) already, so you're really just replicating effort and complexity instead.
- Arnavion 5y ago
  Indeed. k8s orchestration is also useful for running multiple replicas of a database and using leader election to promote a secondary to a primary if the primary goes down. Doing this without k8s would require some kind of clustering anyway, to coordinate the leader election. So if you're already using k8s for everything else, you might as well also use it for this instead of introducing something else like pacemaker. The one disadvantage is that leader election under k8s does need to be implemented from scratch, because it isn't a first-party concept in k8s. All it gives you is the first-party implementation of compare-and-swap for resources. So you have to make a StatefulSet for your replicas, make a ConfigMap to store the leader election data (which is where the CAS ensures atomicity), and write a sidecar container to handle the healthchecks and leader election and dynamic scale-up/down of replicas. You'll probably end up making an operator to do all that for you so that you can deploy a custom resource for your replicas and they're converted to StatefulSets with Pods containing the sidecars, etc automatically. Of course, for the big popular DBs, someone has probably already written such operators for you.
- briffle 5y ago
  what happens when you declare that your PG 13 cluster should now be PG 14? I haven't yet seen good documentation on how these K8s databases handle upgrades. Additionally, they all seem very quick to stream from master to create a new node. That is great, for very, very small db's. But a 1TB db would be royal pain to do that with.
  - bitwalker 5y ago
    I think this is where the operator pattern really shines. By defining a custom resource that contains the cluster configuration, the operator can detect certain types of changes, such as upgrading Postgres to a new major version, and automate that change the same way that you'd do it manually. Of course, if such an operator doesn't already exist, its on you to build it, but with popular databases like Postgres, they are generally already out there in some form. That said, I'm not sure if existing Postgres operators handle your specific example.
dividedbyzero 5y ago
I've ran several small postgres databases and a kafka cluster on Kubernetes; in our case, the VM route would have added lots and lots of overhead with very questionable benefits, given that we didn't run a single prod VM ourselves (all managed by GKE) and hence we had good tooling, observability, monitoring for k8s workloads in place, but VMs were essentially treated as more like a set of disposable blackboxes (not entirely, but we'd only monitor certain aspects and leave all else to GKE) Being able to get essentially the same result, but with our familiar processes, tooling, infra, that was really helpful getting things off the ground quickly and keeping them stable. We picked k8s/GKE way before that as a platform where a small set of abstractions allow us to do a lot of different things, and that makes it easy to debug unfamiliar workloads, keeping operations overhead low, and it worked out really well, and much easier for people with little ops experience to learn and get to a point where they could take care of their workloads themselves. And it never felt we had to mangle k8s for our stateful workloads at all. Statefulsets, storage, node affinity etc. feel very much like organic building blocks and work really really well, with very few surprises. I guess k8s has come a long way as far as stateful workloads are concerned.

manigandham 5y ago

K8S runs workloads across servers in a declarative fashion. Ephemeral or stateful doesn't make a difference anymore. Whatever you use to manage the nodes directly is basically what K8S provides (and more) already, so you're really just replicating effort and complexity instead.

Arnavion 5y ago
Indeed. k8s orchestration is also useful for running multiple replicas of a database and using leader election to promote a secondary to a primary if the primary goes down. Doing this without k8s would require some kind of clustering anyway, to coordinate the leader election. So if you're already using k8s for everything else, you might as well also use it for this instead of introducing something else like pacemaker. The one disadvantage is that leader election under k8s does need to be implemented from scratch, because it isn't a first-party concept in k8s. All it gives you is the first-party implementation of compare-and-swap for resources. So you have to make a StatefulSet for your replicas, make a ConfigMap to store the leader election data (which is where the CAS ensures atomicity), and write a sidecar container to handle the healthchecks and leader election and dynamic scale-up/down of replicas. You'll probably end up making an operator to do all that for you so that you can deploy a custom resource for your replicas and they're converted to StatefulSets with Pods containing the sidecars, etc automatically. Of course, for the big popular DBs, someone has probably already written such operators for you.
briffle 5y ago
what happens when you declare that your PG 13 cluster should now be PG 14? I haven't yet seen good documentation on how these K8s databases handle upgrades. Additionally, they all seem very quick to stream from master to create a new node. That is great, for very, very small db's. But a 1TB db would be royal pain to do that with.
- bitwalker 5y ago
  I think this is where the operator pattern really shines. By defining a custom resource that contains the cluster configuration, the operator can detect certain types of changes, such as upgrading Postgres to a new major version, and automate that change the same way that you'd do it manually. Of course, if such an operator doesn't already exist, its on you to build it, but with popular databases like Postgres, they are generally already out there in some form. That said, I'm not sure if existing Postgres operators handle your specific example.

dividedbyzero 5y ago

I've ran several small postgres databases and a kafka cluster on Kubernetes; in our case, the VM route would have added lots and lots of overhead with very questionable benefits, given that we didn't run a single prod VM ourselves (all managed by GKE) and hence we had good tooling, observability, monitoring for k8s workloads in place, but VMs were essentially treated as more like a set of disposable blackboxes (not entirely, but we'd only monitor certain aspects and leave all else to GKE) Being able to get essentially the same result, but with our familiar processes, tooling, infra, that was really helpful getting things off the ground quickly and keeping them stable. We picked k8s/GKE way before that as a platform where a small set of abstractions allow us to do a lot of different things, and that makes it easy to debug unfamiliar workloads, keeping operations overhead low, and it worked out really well, and much easier for people with little ops experience to learn and get to a point where they could take care of their workloads themselves. And it never felt we had to mangle k8s for our stateful workloads at all. Statefulsets, storage, node affinity etc. feel very much like organic building blocks and work really really well, with very few surprises. I guess k8s has come a long way as far as stateful workloads are concerned.