4 ms·

StatefulSet and PVCs aren’t sufficient to fully handle all the likely resilience challenges of running a database cluster on K8S. There needs to be some rethink

StatefulSet and PVCs aren’t sufficient to fully handle all the likely resilience challenges of running a database cluster on K8S. There needs to be some rethinking on how StatefulSet works to make it more appropriate to this use case, such as allowing Pods to be started out of order when recovering from failures. I worked in this problem space extensively until 2020, and I think that there are paths forward but they require changes in K8S that none of the folks involved seem motivated to make. Realistically to make databases in K8S work well today you need a database built for K8S rather than one adapted for K8S. The building blocks present today are not fundamentally capable of building a positive UX for adapting existing databases to K8S, but this is something that is worth making possible and I hope the community gets there some day.

smarterclayton 4y ago
Re out of order: Is https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#parallel-pod-management https://kubernetes.io/docs/concepts/workloads/controllers/st... unsuitable for that?
- tristor 4y ago
  Unfortunately, while rolling updates account for some scenarios, they are not sufficient for handling out of order restarts where the order cannot be pre-determined. There’s probably some hack you could build with partitioning to mostly address the cases I am thinking of, but it isn’t elegant or guaranteed correct. This will be a problem for any database where clustering is synchronous and a specific primary node must start first on a full cluster restart. There are other out of band hacks you can do with reassigning PVCs, but it’s never elegant in the current primitives provided. During my work in this problem space I became convinced that primitives for stateful applications in K8S were built specifically without considering databases as a valid use case. Everything else is just hacks after the fact to make it “work”.
  - smarterclayton 4y ago
    Since I helped design them, I take some issue with that :). Certainly we never expected they would completely solve problems for the database, but they were definitely intended to provide guarantees that simplify normal consensus operations and prevent accidental confusion with non perfect databases. If a specific primary must start first, that’s partially what ordinals were intended to allow (0 is your primary, the others are always 1-N, and kube is responsible for ensuring the primary is never reassigned). I’d love to take feedback about places where the primitives are unusable, or ways they can be improved, because there are always new tools to add.
    - tristor 4y ago
      I am not working in this problem space any longer, but it’s likely we’ve crossed paths. I have previously presented some suggestions to the Storage SIG on this topic. Feel free to reach out to me with the info in my profile and I can get back to you with a more detailed write up on the specific challenges that I would like to see addressed in StatefulSet, however I am currently traveling so my response will be delayed.
      - bogomipz 4y ago
        >"I have previously presented some suggestions to the Storage SIG on this topic." Ia there any chance you might be able to provide a link to these? I would be curious to take a look.
  - throwdbaaway 4y ago
    You are talking about things like the "All Nodes Go Down Without a Proper Shutdown Procedure" example in https://galeracluster.com/library/documentation/crash-recovery.html https://galeracluster.com/library/documentation/crash-recove... right? To handle this case, some teams may have a manual runbook, some teams may have some automation with ansible, some teams may have nothing. So, if someone can come up with some hacks and package that into a k8s operator, it is still a win. It seems to be the best primitives we have at the moment.
bogomipz 4y ago
>"I worked in this problem space extensively until 2020, and I think that there are paths forward but they require changes in K8S that none of the folks involved seem motivated to make." Can you say what you see as those possible paths forward and what changes they would require?

smarterclayton 4y ago

Re out of order: Is https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#parallel-pod-management https://kubernetes.io/docs/concepts/workloads/controllers/st... unsuitable for that?

tristor 4y ago
Unfortunately, while rolling updates account for some scenarios, they are not sufficient for handling out of order restarts where the order cannot be pre-determined. There’s probably some hack you could build with partitioning to mostly address the cases I am thinking of, but it isn’t elegant or guaranteed correct. This will be a problem for any database where clustering is synchronous and a specific primary node must start first on a full cluster restart. There are other out of band hacks you can do with reassigning PVCs, but it’s never elegant in the current primitives provided. During my work in this problem space I became convinced that primitives for stateful applications in K8S were built specifically without considering databases as a valid use case. Everything else is just hacks after the fact to make it “work”.
- smarterclayton 4y ago
  Since I helped design them, I take some issue with that :). Certainly we never expected they would completely solve problems for the database, but they were definitely intended to provide guarantees that simplify normal consensus operations and prevent accidental confusion with non perfect databases. If a specific primary must start first, that’s partially what ordinals were intended to allow (0 is your primary, the others are always 1-N, and kube is responsible for ensuring the primary is never reassigned). I’d love to take feedback about places where the primitives are unusable, or ways they can be improved, because there are always new tools to add.
  - tristor 4y ago
    I am not working in this problem space any longer, but it’s likely we’ve crossed paths. I have previously presented some suggestions to the Storage SIG on this topic. Feel free to reach out to me with the info in my profile and I can get back to you with a more detailed write up on the specific challenges that I would like to see addressed in StatefulSet, however I am currently traveling so my response will be delayed.
    - bogomipz 4y ago
      >"I have previously presented some suggestions to the Storage SIG on this topic." Ia there any chance you might be able to provide a link to these? I would be curious to take a look.
- throwdbaaway 4y ago
  You are talking about things like the "All Nodes Go Down Without a Proper Shutdown Procedure" example in https://galeracluster.com/library/documentation/crash-recovery.html https://galeracluster.com/library/documentation/crash-recove... right? To handle this case, some teams may have a manual runbook, some teams may have some automation with ansible, some teams may have nothing. So, if someone can come up with some hacks and package that into a k8s operator, it is still a win. It seems to be the best primitives we have at the moment.

bogomipz 4y ago

>"I worked in this problem space extensively until 2020, and I think that there are paths forward but they require changes in K8S that none of the folks involved seem motivated to make." Can you say what you see as those possible paths forward and what changes they would require?