8 ms·

Say you have 100 database clusters and you want to schedule them on dedicated servers. A kubernetes node can advertise those local block devices as persistent

Say you have 100 database clusters and you want to schedule them on dedicated servers. A kubernetes node can advertise those local block devices as persistent volumes. Your StatefulSet can request that class of local storage and the kubernetes scheduler will then find a volume for the pods on a node that the pod can fit on. You then get all the features of kubernetes operators plus the performance of dedicated nodes and local storage. When a local storage node disappears, your pod gets a new volume on a new node and can replicate from its peers, or start from a backup. Kubernetes then offers metrics, certificate management, secret storage and management, load balancing, healthchecks, debugging tools, rich firewalling, RBAC, and a huge ecosystem of plugins. It gives you a unified API for all your workloads. Requesting a database then is as simple as a single API call to kubernetes with a spec for your database (ex: creating a MysqlCluster object). The company I work for runs MySQL, ZooKeeper, memcached, hbase, hadoop, kafka, elasticsearch, and more in kubernetes with great success. Scale here in one prod region is ~10,000 pods on a clusters of ~600 nodes. There are about 900 separate MySQL clusters consisting of 3 replicas each, 150 separate zk clusters of 5 replicas each, etc.

arianvanp 5y ago
> when a node disappears you get a new volume Actually no. Currently you need to manually delete the persistent volume claim. Followed by deleting the pod. For the statefulsets ti to create a new PVC. There even was a race condition until like 1.21 that made this operation fail 20% of the time so you'd have to delete the pod twice or the system would deadlock. It'd be nice if the PVC would claim a new PVC if the orchestration layer knows 100% the PV is gone but it currently doesn't do this yet. It's not an enormous problem once you're aware of it. But it does mean things don't auto heal at all. A node going down with local storage on it needs manual intervention to get the system running and healthy again. There might be some operators that automate this away but i haven't seen it yet in the wild.
- paulfurtado 5y ago
  We do actually automate this away by deleting PVCs when they are tied to PVs on nodes with our drain taint on them and their pod is in the pending state. We also delete pods owned my StatefulSets that have missing PVCs. These two control loops really grease the wheels. It's definitely frustrating that statefulset pvc management isn't fully declarative, but hopefully this improves.
  - chousuke 5y ago
    How do you prevent someone (or something) from accidentally draining a node hosting a database that does not have a synchronised replica? Kubernetes is neat, but destroying stuff is so easy (and the context mechanism is just begging for human error) that I am a bit leery about hosting large amounts of stateful stuff in it.
    - paulfurtado 5y ago
      No database should lack an unsynchronized replica in general. A kubernetes admin could accidentally drain the host, but we run in AWS and AWS can kill nodes a lot faster than we can haha. We have enough servers in AWS that at least 5 fail per day. If some database is not replicated, someone is getting paged for it. We run MySQL with semi-sync replication for example: if a primary has two replicas, one replica must ack a write before the primary commits it, so you can lose the primary at any time and not lose data. That said, kubernetes has prevention mechanisms for this. PodDisruptionBudgets allow you to specify the minimum number of replicas. The drain command calls evict instead of delete and evict honors these restrictions. This is helpful for node draining, but not necessarily helpful if someone accidentally deletes a StatefulSet. However, nothing in kubernetes deletes PVC objects automatically, so if you delete the pod, the volume remains bound to the PVC until you delete it. preStop hooks are used in all the database pods to ensure that if the current pod is the primary, leadership is correctly changed to a replica before proceeding with the delete. We combine this with extremely long termination grace periods and alerting if termination is taking too long so that an ops team can look into the issue. preStop hooks are great, but they also are somewhat of a time bomb if not careful, so some of our database operators make use of finalizers instead. With finalizers, nothing happens when a pod enters the terminating state until an operator marks the pod's finalizer as completed. This flow is really nice because you end up with behavior where the kubernetes cluster admins or built in controllers effectively are really gently requesting that a pod terminate, and it's up to the operator of that pod to decide exactly how and when that occurs. For non-local volumes, like EBS PVCs, once the PVC is deleted an operator will snapshot the volume during a finalizer before allowing the volume to be fully deleted so that we can recover data if this ever accidentally occurs. This was the first protection we ever implemented when initially adopting kubernetes back when we had no idea what we were doing so that we could always recover data in the event of admin mistakes. We can't do this for local volumes, however, any team making use of local volumes truly needs to stay on top of replication when running in the cloud, whether on kubernetes or not or else you absolutely will lose data. AWS local nvme volumes have limited write cycles just like the ones in on-prem servers. So ideally all databses are configured to not accept writes that cannot be replicated. And even with EBS, at scale we end up with something like 20 unrecoverable volume failures per year

arianvanp 5y ago

> when a node disappears you get a new volume Actually no. Currently you need to manually delete the persistent volume claim. Followed by deleting the pod. For the statefulsets ti to create a new PVC. There even was a race condition until like 1.21 that made this operation fail 20% of the time so you'd have to delete the pod twice or the system would deadlock. It'd be nice if the PVC would claim a new PVC if the orchestration layer knows 100% the PV is gone but it currently doesn't do this yet. It's not an enormous problem once you're aware of it. But it does mean things don't auto heal at all. A node going down with local storage on it needs manual intervention to get the system running and healthy again. There might be some operators that automate this away but i haven't seen it yet in the wild.

paulfurtado 5y ago
We do actually automate this away by deleting PVCs when they are tied to PVs on nodes with our drain taint on them and their pod is in the pending state. We also delete pods owned my StatefulSets that have missing PVCs. These two control loops really grease the wheels. It's definitely frustrating that statefulset pvc management isn't fully declarative, but hopefully this improves.
- chousuke 5y ago
  How do you prevent someone (or something) from accidentally draining a node hosting a database that does not have a synchronised replica? Kubernetes is neat, but destroying stuff is so easy (and the context mechanism is just begging for human error) that I am a bit leery about hosting large amounts of stateful stuff in it.
  - paulfurtado 5y ago
    No database should lack an unsynchronized replica in general. A kubernetes admin could accidentally drain the host, but we run in AWS and AWS can kill nodes a lot faster than we can haha. We have enough servers in AWS that at least 5 fail per day. If some database is not replicated, someone is getting paged for it. We run MySQL with semi-sync replication for example: if a primary has two replicas, one replica must ack a write before the primary commits it, so you can lose the primary at any time and not lose data. That said, kubernetes has prevention mechanisms for this. PodDisruptionBudgets allow you to specify the minimum number of replicas. The drain command calls evict instead of delete and evict honors these restrictions. This is helpful for node draining, but not necessarily helpful if someone accidentally deletes a StatefulSet. However, nothing in kubernetes deletes PVC objects automatically, so if you delete the pod, the volume remains bound to the PVC until you delete it. preStop hooks are used in all the database pods to ensure that if the current pod is the primary, leadership is correctly changed to a replica before proceeding with the delete. We combine this with extremely long termination grace periods and alerting if termination is taking too long so that an ops team can look into the issue. preStop hooks are great, but they also are somewhat of a time bomb if not careful, so some of our database operators make use of finalizers instead. With finalizers, nothing happens when a pod enters the terminating state until an operator marks the pod's finalizer as completed. This flow is really nice because you end up with behavior where the kubernetes cluster admins or built in controllers effectively are really gently requesting that a pod terminate, and it's up to the operator of that pod to decide exactly how and when that occurs. For non-local volumes, like EBS PVCs, once the PVC is deleted an operator will snapshot the volume during a finalizer before allowing the volume to be fully deleted so that we can recover data if this ever accidentally occurs. This was the first protection we ever implemented when initially adopting kubernetes back when we had no idea what we were doing so that we could always recover data in the event of admin mistakes. We can't do this for local volumes, however, any team making use of local volumes truly needs to stay on top of replication when running in the cloud, whether on kubernetes or not or else you absolutely will lose data. AWS local nvme volumes have limited write cycles just like the ones in on-prem servers. So ideally all databses are configured to not accept writes that cannot be replicated. And even with EBS, at scale we end up with something like 20 unrecoverable volume failures per year