3 ms·

Running a statefull service in K8S is its own ball of wax

by wbl 4y ago

foobarian 4y ago
Yes, Postgres on K8S... <shudder>
patrec 4y ago
It is, but then I never understood why on earth you'd use k8s if you don't have stateful services. I mean really, what's the point?
- mijamo 4y ago
  Because it's easy? What alternative would you suggest?
  - janee 4y ago
    Bare metal I'd think is the first choice for a large rdbms where you have skilled dedicated personnel that can manage it. If not rather use a specialist service like RDS for anything with serious uptime/throughput requirements. k8s doesn't really make sense to me unless it's for spinning up lots of instances, like for test or dev envs or like in the article where they host DBs for people.
  - patrec 4y ago
    The idea that something of the monstrous complexity of k8s is easy is pretty funny to me. I think if you have less than than 2 full time experts on k8s at hand, you're basically nuts if you use it for some non-toy project. In my experience, you can and will experience interesting failure scenarios. If you don't have state, why not just either use something serverless/fully-managed (beanstalk, lambda, cloudflare workers whatever) if you really need to scale up and down (or have very limited devops/sysadmin capacity) or deploy like 2 or 3 bare metal machines or VMs? Either sounds like a lot less work to manage and troubleshoot than some freaking k8s cluster.
deathanatos 4y ago
… I do it, in my day job. It's really not. StatefulSets are explicitly for this. We also have managed databases, too. Self-managed stuff means I can, generally, get shit done with it, when oddball things need doing. Managed stuff is fine right up until it isn't (i.e., yet another outage with the status page being green), or until there's a requirement that the managed system inexplicably can't handle (despite the requirement being the sort of obvious thing you would expect of $SYSTEM, but which no PM thought to ask before purchasing the deal…), and then you're in support ticket hell. (E.g., we found out the hard way that there is not way to move a managed PG database from one subnet in a network to another, in Azure! Even if you're willing to restore from a backup. We had to deal with that ourselves, by taking a pgdump — essentially, un-managed-solution the backup. … the whole reason we needed to move the DB to a different subnet was because of a different flaw, in a different managed service, and Azure's answer on that ticket was "tough luck DB needs to move". Tickets, spawning tickets. Support tickets for managed services take up an unholy portion of my time.)

Running a statefull service in K8S is its own ball of wax

by wbl 4y ago

Running a statefull service in K8S is its own ball of wax

foobarian 4y ago

Yes, Postgres on K8S... <shudder>

patrec 4y ago

It is, but then I never understood why on earth you'd use k8s if you don't have stateful services. I mean really, what's the point?

mijamo 4y ago
Because it's easy? What alternative would you suggest?
- janee 4y ago
  Bare metal I'd think is the first choice for a large rdbms where you have skilled dedicated personnel that can manage it. If not rather use a specialist service like RDS for anything with serious uptime/throughput requirements. k8s doesn't really make sense to me unless it's for spinning up lots of instances, like for test or dev envs or like in the article where they host DBs for people.
- patrec 4y ago
  The idea that something of the monstrous complexity of k8s is easy is pretty funny to me. I think if you have less than than 2 full time experts on k8s at hand, you're basically nuts if you use it for some non-toy project. In my experience, you can and will experience interesting failure scenarios. If you don't have state, why not just either use something serverless/fully-managed (beanstalk, lambda, cloudflare workers whatever) if you really need to scale up and down (or have very limited devops/sysadmin capacity) or deploy like 2 or 3 bare metal machines or VMs? Either sounds like a lot less work to manage and troubleshoot than some freaking k8s cluster.

deathanatos 4y ago

… I do it, in my day job. It's really not. StatefulSets are explicitly for this. We also have managed databases, too. Self-managed stuff means I can, generally, get shit done with it, when oddball things need doing. Managed stuff is fine right up until it isn't (i.e., yet another outage with the status page being green), or until there's a requirement that the managed system inexplicably can't handle (despite the requirement being the sort of obvious thing you would expect of $SYSTEM, but which no PM thought to ask before purchasing the deal…), and then you're in support ticket hell. (E.g., we found out the hard way that there is not way to move a managed PG database from one subnet in a network to another, in Azure! Even if you're willing to restore from a backup. We had to deal with that ourselves, by taking a pgdump — essentially, un-managed-solution the backup. … the whole reason we needed to move the DB to a different subnet was because of a different flaw, in a different managed service, and Azure's answer on that ticket was "tough luck DB needs to move". Tickets, spawning tickets. Support tickets for managed services take up an unholy portion of my time.)