5 ms·

With a database I generally want to have it on its own dedicated host with fast NVME drives, memory, cpu etc, maybe some OS specific system level configuration

With a database I generally want to have it on its own dedicated host with fast NVME drives, memory, cpu etc, maybe some OS specific system level configuration options for performance etc. I don't want to binpack it on a cluster of nodes each contending for system resources. Also RDBMS databases are not really designed to be ephemeral/elastic, they are stateful/persistent by nature. What does Kubernetes offer in this scenario? I'm still hesitant to run things like DB's on K8's but have absolutely no problems running cloud native apps on K8's which can run ephemerally, be bin packed and only needing compute as all state is external to where the app is being executed.

alphabettsy 5y ago
StatefulSets, Node Groups, and Node Affinity/Taints solve some of those issues.
- karolist 5y ago
  A database is not a service well suited for ephemeral, quick scale down and scale up container workloads that K8s really excells at, and if you have nodes specifically configured to run databases and nothing else then you should simply run them there directly and not mangle K8s to replicate what will essentially be equivalent to running them there directly.
  - manigandham 5y ago
    K8S runs workloads across servers in a declarative fashion. Ephemeral or stateful doesn't make a difference anymore. Whatever you use to manage the nodes directly is basically what K8S provides (and more) already, so you're really just replicating effort and complexity instead.
    - Arnavion 5y ago
      Indeed. k8s orchestration is also useful for running multiple replicas of a database and using leader election to promote a secondary to a primary if the primary goes down. Doing this without k8s would require some kind of clustering anyway, to coordinate the leader election. So if you're already using k8s for everything else, you might as well also use it for this instead of introducing something else like pacemaker. The one disadvantage is that leader election under k8s does need to be implemented from scratch, because it isn't a first-party concept in k8s. All it gives you is the first-party implementation of compare-and-swap for resources. So you have to make a StatefulSet for your replicas, make a ConfigMap to store the leader election data (which is where the CAS ensures atomicity), and write a sidecar container to handle the healthchecks and leader election and dynamic scale-up/down of replicas. You'll probably end up making an operator to do all that for you so that you can deploy a custom resource for your replicas and they're converted to StatefulSets with Pods containing the sidecars, etc automatically. Of course, for the big popular DBs, someone has probably already written such operators for you.
    - briffle 5y ago
      what happens when you declare that your PG 13 cluster should now be PG 14? I haven't yet seen good documentation on how these K8s databases handle upgrades. Additionally, they all seem very quick to stream from master to create a new node. That is great, for very, very small db's. But a 1TB db would be royal pain to do that with.
      - bitwalker 5y ago
        I think this is where the operator pattern really shines. By defining a custom resource that contains the cluster configuration, the operator can detect certain types of changes, such as upgrading Postgres to a new major version, and automate that change the same way that you'd do it manually. Of course, if such an operator doesn't already exist, its on you to build it, but with popular databases like Postgres, they are generally already out there in some form. That said, I'm not sure if existing Postgres operators handle your specific example.
  - dividedbyzero 5y ago
    I've ran several small postgres databases and a kafka cluster on Kubernetes; in our case, the VM route would have added lots and lots of overhead with very questionable benefits, given that we didn't run a single prod VM ourselves (all managed by GKE) and hence we had good tooling, observability, monitoring for k8s workloads in place, but VMs were essentially treated as more like a set of disposable blackboxes (not entirely, but we'd only monitor certain aspects and leave all else to GKE) Being able to get essentially the same result, but with our familiar processes, tooling, infra, that was really helpful getting things off the ground quickly and keeping them stable. We picked k8s/GKE way before that as a platform where a small set of abstractions allow us to do a lot of different things, and that makes it easy to debug unfamiliar workloads, keeping operations overhead low, and it worked out really well, and much easier for people with little ops experience to learn and get to a point where they could take care of their workloads themselves. And it never felt we had to mangle k8s for our stateful workloads at all. Statefulsets, storage, node affinity etc. feel very much like organic building blocks and work really really well, with very few surprises. I guess k8s has come a long way as far as stateful workloads are concerned.
- funcDropShadow 5y ago
  So, the argument to deploy a database (e.g. Postgres) via Kubernetes is, that that some of the problems doing so are now solved?
FridgeSeal 5y ago
As the other comment or pointed out, there’s a number of features now that tick these boxes: - StatefulSets mean that K8s scheduling is aware that the workload is stateful and the auto-attach of drives works fantastically in my experience. - NodeTaints/Tolerances and workload anti-affinities between workloads mean that you can very easily say “this workload runs on nodes with these features/labels” and “don’t run workload replicas on the same node” and “don’t run other workloads on the same node as this workload. Node groups on things like AWS EKS also make it really easy to get autoscaling groups of nodes with specific hardware.
paulfurtado 5y ago
Say you have 100 database clusters and you want to schedule them on dedicated servers. A kubernetes node can advertise those local block devices as persistent volumes. Your StatefulSet can request that class of local storage and the kubernetes scheduler will then find a volume for the pods on a node that the pod can fit on. You then get all the features of kubernetes operators plus the performance of dedicated nodes and local storage. When a local storage node disappears, your pod gets a new volume on a new node and can replicate from its peers, or start from a backup. Kubernetes then offers metrics, certificate management, secret storage and management, load balancing, healthchecks, debugging tools, rich firewalling, RBAC, and a huge ecosystem of plugins. It gives you a unified API for all your workloads. Requesting a database then is as simple as a single API call to kubernetes with a spec for your database (ex: creating a MysqlCluster object). The company I work for runs MySQL, ZooKeeper, memcached, hbase, hadoop, kafka, elasticsearch, and more in kubernetes with great success. Scale here in one prod region is ~10,000 pods on a clusters of ~600 nodes. There are about 900 separate MySQL clusters consisting of 3 replicas each, 150 separate zk clusters of 5 replicas each, etc.
- arianvanp 5y ago
  > when a node disappears you get a new volume Actually no. Currently you need to manually delete the persistent volume claim. Followed by deleting the pod. For the statefulsets ti to create a new PVC. There even was a race condition until like 1.21 that made this operation fail 20% of the time so you'd have to delete the pod twice or the system would deadlock. It'd be nice if the PVC would claim a new PVC if the orchestration layer knows 100% the PV is gone but it currently doesn't do this yet. It's not an enormous problem once you're aware of it. But it does mean things don't auto heal at all. A node going down with local storage on it needs manual intervention to get the system running and healthy again. There might be some operators that automate this away but i haven't seen it yet in the wild.
  - paulfurtado 5y ago
    We do actually automate this away by deleting PVCs when they are tied to PVs on nodes with our drain taint on them and their pod is in the pending state. We also delete pods owned my StatefulSets that have missing PVCs. These two control loops really grease the wheels. It's definitely frustrating that statefulset pvc management isn't fully declarative, but hopefully this improves.
arianvanp 5y ago
Have a look at local storage provisioners. There are plenty of them They give you raw access to storage without redundancy. Which is great if your workload on top (like https://k8ssandra.io https://k8ssandra.io) already is redundant. Whilst making sure workloads stick to where their data is stored E.g. https://github.com/openebs/lvm-localpv https://github.com/openebs/lvm-localpv
manigandham 5y ago
Kubernetes is an orchestrator: You have servers. One of them is the master, the rest are workers. Master talks to the workers via ssh. You talk to the master via API. You tell the master node what you want running and it runs it for you. Then it continuously checks to make sure that what's actually running is what you wanted, and makes fixes if it isn't. What you want to run, and how, where, under what conditions, etc. is all specified in declarative files. Like I said, stateful services are not a problem anymore. K8S can deploy a database service as instances exclusive to specific nodes with the appropriate storage and networking config. It's like an invisible assistant to automate what you would be doing yourself, or using several other tools to do anyway. A well-tested way to handle most of the common infrastructure issues for any deployed software in a single platform. EDIT: I'll add that running K8S itself can be challenging, but that's because complexity can't be erased but only abstracted. I recommend using managed K8S to offload this.
prpl 5y ago
minio’s DirectCSI might be a solution you are interested in

alphabettsy 5y ago

StatefulSets, Node Groups, and Node Affinity/Taints solve some of those issues.

karolist 5y ago
A database is not a service well suited for ephemeral, quick scale down and scale up container workloads that K8s really excells at, and if you have nodes specifically configured to run databases and nothing else then you should simply run them there directly and not mangle K8s to replicate what will essentially be equivalent to running them there directly.
- manigandham 5y ago
  K8S runs workloads across servers in a declarative fashion. Ephemeral or stateful doesn't make a difference anymore. Whatever you use to manage the nodes directly is basically what K8S provides (and more) already, so you're really just replicating effort and complexity instead.
  - Arnavion 5y ago
    Indeed. k8s orchestration is also useful for running multiple replicas of a database and using leader election to promote a secondary to a primary if the primary goes down. Doing this without k8s would require some kind of clustering anyway, to coordinate the leader election. So if you're already using k8s for everything else, you might as well also use it for this instead of introducing something else like pacemaker. The one disadvantage is that leader election under k8s does need to be implemented from scratch, because it isn't a first-party concept in k8s. All it gives you is the first-party implementation of compare-and-swap for resources. So you have to make a StatefulSet for your replicas, make a ConfigMap to store the leader election data (which is where the CAS ensures atomicity), and write a sidecar container to handle the healthchecks and leader election and dynamic scale-up/down of replicas. You'll probably end up making an operator to do all that for you so that you can deploy a custom resource for your replicas and they're converted to StatefulSets with Pods containing the sidecars, etc automatically. Of course, for the big popular DBs, someone has probably already written such operators for you.
  - briffle 5y ago
    what happens when you declare that your PG 13 cluster should now be PG 14? I haven't yet seen good documentation on how these K8s databases handle upgrades. Additionally, they all seem very quick to stream from master to create a new node. That is great, for very, very small db's. But a 1TB db would be royal pain to do that with.
    - bitwalker 5y ago
      I think this is where the operator pattern really shines. By defining a custom resource that contains the cluster configuration, the operator can detect certain types of changes, such as upgrading Postgres to a new major version, and automate that change the same way that you'd do it manually. Of course, if such an operator doesn't already exist, its on you to build it, but with popular databases like Postgres, they are generally already out there in some form. That said, I'm not sure if existing Postgres operators handle your specific example.
- dividedbyzero 5y ago
  I've ran several small postgres databases and a kafka cluster on Kubernetes; in our case, the VM route would have added lots and lots of overhead with very questionable benefits, given that we didn't run a single prod VM ourselves (all managed by GKE) and hence we had good tooling, observability, monitoring for k8s workloads in place, but VMs were essentially treated as more like a set of disposable blackboxes (not entirely, but we'd only monitor certain aspects and leave all else to GKE) Being able to get essentially the same result, but with our familiar processes, tooling, infra, that was really helpful getting things off the ground quickly and keeping them stable. We picked k8s/GKE way before that as a platform where a small set of abstractions allow us to do a lot of different things, and that makes it easy to debug unfamiliar workloads, keeping operations overhead low, and it worked out really well, and much easier for people with little ops experience to learn and get to a point where they could take care of their workloads themselves. And it never felt we had to mangle k8s for our stateful workloads at all. Statefulsets, storage, node affinity etc. feel very much like organic building blocks and work really really well, with very few surprises. I guess k8s has come a long way as far as stateful workloads are concerned.
funcDropShadow 5y ago
So, the argument to deploy a database (e.g. Postgres) via Kubernetes is, that that some of the problems doing so are now solved?

FridgeSeal 5y ago

As the other comment or pointed out, there’s a number of features now that tick these boxes: - StatefulSets mean that K8s scheduling is aware that the workload is stateful and the auto-attach of drives works fantastically in my experience. - NodeTaints/Tolerances and workload anti-affinities between workloads mean that you can very easily say “this workload runs on nodes with these features/labels” and “don’t run workload replicas on the same node” and “don’t run other workloads on the same node as this workload. Node groups on things like AWS EKS also make it really easy to get autoscaling groups of nodes with specific hardware.

paulfurtado 5y ago

Say you have 100 database clusters and you want to schedule them on dedicated servers. A kubernetes node can advertise those local block devices as persistent volumes. Your StatefulSet can request that class of local storage and the kubernetes scheduler will then find a volume for the pods on a node that the pod can fit on. You then get all the features of kubernetes operators plus the performance of dedicated nodes and local storage. When a local storage node disappears, your pod gets a new volume on a new node and can replicate from its peers, or start from a backup. Kubernetes then offers metrics, certificate management, secret storage and management, load balancing, healthchecks, debugging tools, rich firewalling, RBAC, and a huge ecosystem of plugins. It gives you a unified API for all your workloads. Requesting a database then is as simple as a single API call to kubernetes with a spec for your database (ex: creating a MysqlCluster object). The company I work for runs MySQL, ZooKeeper, memcached, hbase, hadoop, kafka, elasticsearch, and more in kubernetes with great success. Scale here in one prod region is ~10,000 pods on a clusters of ~600 nodes. There are about 900 separate MySQL clusters consisting of 3 replicas each, 150 separate zk clusters of 5 replicas each, etc.

arianvanp 5y ago
> when a node disappears you get a new volume Actually no. Currently you need to manually delete the persistent volume claim. Followed by deleting the pod. For the statefulsets ti to create a new PVC. There even was a race condition until like 1.21 that made this operation fail 20% of the time so you'd have to delete the pod twice or the system would deadlock. It'd be nice if the PVC would claim a new PVC if the orchestration layer knows 100% the PV is gone but it currently doesn't do this yet. It's not an enormous problem once you're aware of it. But it does mean things don't auto heal at all. A node going down with local storage on it needs manual intervention to get the system running and healthy again. There might be some operators that automate this away but i haven't seen it yet in the wild.
- paulfurtado 5y ago
  We do actually automate this away by deleting PVCs when they are tied to PVs on nodes with our drain taint on them and their pod is in the pending state. We also delete pods owned my StatefulSets that have missing PVCs. These two control loops really grease the wheels. It's definitely frustrating that statefulset pvc management isn't fully declarative, but hopefully this improves.

arianvanp 5y ago

Have a look at local storage provisioners. There are plenty of them They give you raw access to storage without redundancy. Which is great if your workload on top (like https://k8ssandra.io https://k8ssandra.io) already is redundant. Whilst making sure workloads stick to where their data is stored E.g. https://github.com/openebs/lvm-localpv https://github.com/openebs/lvm-localpv

manigandham 5y ago

Kubernetes is an orchestrator: You have servers. One of them is the master, the rest are workers. Master talks to the workers via ssh. You talk to the master via API. You tell the master node what you want running and it runs it for you. Then it continuously checks to make sure that what's actually running is what you wanted, and makes fixes if it isn't. What you want to run, and how, where, under what conditions, etc. is all specified in declarative files. Like I said, stateful services are not a problem anymore. K8S can deploy a database service as instances exclusive to specific nodes with the appropriate storage and networking config. It's like an invisible assistant to automate what you would be doing yourself, or using several other tools to do anyway. A well-tested way to handle most of the common infrastructure issues for any deployed software in a single platform. EDIT: I'll add that running K8S itself can be challenging, but that's because complexity can't be erased but only abstracted. I recommend using managed K8S to offload this.

prpl 5y ago

minio’s DirectCSI might be a solution you are interested in