5 ms·

Just like with cloud providers, it seems to me like with kubernetes, it is not a matter of if but when orchestration problems will arise. Specially in this case

Just like with cloud providers, it seems to me like with kubernetes, it is not a matter of if but when orchestration problems will arise. Specially in this case of hosting databases, a composition of provisioning complexities (db operational complexities on top of k8s operational complexities) is really scary. Is there any way to overcome hidden complexity biting your hand other than studying k8s extensively?

renegade-otter 3y ago
This kind of resilience is a form of art, and it's also kind of a full-time job. I would not advise trying this for a "side project at work". Generally we all agree that we move to the cloud and it's "fully managed". If it goes down - that's the price we pay. If you have to ask the question "what if the RDS goes down", then you are really in a different universe. That last guarantee of uptime requires a ton of work, testing, and money, because you are all the way up and to the right on the curve of diminishing returns.
- candiddevmike 3y ago
  > That last guarantee of uptime requires a ton of work, testing, and money, because you are all the way up and to the right on the curve of diminishing returns. Not when it's a core part of your business...?
- williamdclt 3y ago
  > If you have to ask the question "what if the RDS goes down", then you are really in a different universe. It does go down though, don’t neglect the possibility because it likely will happen. With very average workloads, I’ve seen RDS databases restart unexpectedly, read replicas being completely out of service, and even databases being completely frozen (can’t even connect as root). I’d still go with managed, but it certainly doesn’t give full reliability :) you still have to consider “what if it goes down” - it will!
  - debarshri 3y ago
    Problem is that RDS comes at a price. It is purely about operation cost. When you have 1500+ databases these cost add up. At that point, this kind of techniques are required to self host the databases. Price per DB with HPA na VPA is way lower than what you would pay for managed databases as well as you can hire a full time devops+dbadmin and still be cheaper.
    - SlightlyLeftPad 3y ago
      This is a perfect example of a cost savings opportunity that offers no benefit and is really just a foot gun when it comes to operational complexity. The cloud cost savings almost never justify the operational costs here. There’s typically a very long list of other cost optimizations to be made before this would ever be on the table for me. If you’re talking about 1500+ databases and think a single full time devops+dbadmin in some kind of unicorn person is going to be a better option, I feel bad for both of you.
      - p_l 3y ago
        Except the typical cloud prices are dialed for USA, and it's not that unusual in my experience for the cloud savings to be high enough to add more than one highly skilled engineer who will give large positive impact all around.
      - debarshri 3y ago
        I am not implying hiring single full time dbadmin, you can hire a team of them and still be cheaper that what AWS would cost at that scale.
- figassis 3y ago
  My company also runs on a k8s cluster, but I agree with this fully and I keep my DB outside the cluster. For me, the order of preference is: RDS, Cloud Instance, Bare metal DB server, something else that is not kubernetes. RDS will fail, but it likely will come back up without much action on my part. K8s will fail, you will spend untold human hours figuring out the k8s failure modes, before figuring out the database failure modes (which likely are quite straight forward). It's just a cost that is not worth it.
szszrk 3y ago
Not really. Those are two areas you need to really understand when things go south. In Kubernetes you "solve" a lot of application complexity by abstracting it into things like Helm Charts or Operators. It's fun and easy to use operators to deploy databases, monitoring, mesh, minio... This does not make complexity disappear, it just abstracts it by creating another layer with an interface (k8s API) you may be more familiar with. Whole Kubernetes in practice is adding more and more layers on top of it. All those layers/tools are usually amazing but soon you need some real k8s skills AND some app skills to handle even unavoidable scenarios like ... upgrades. That's why it's so confusing sometimes and has such diverse opinions on. There aren't many environments that run vanilla cluster...
levkk 3y ago
In the case of running Postgres on K8s, the problem arises immediately when you try to resize a data volume and you can't because the API doesn't support it. K8s is not really for stateful systems, yet, and systems like Postgres that prefer to manage their own resources, you don't want another layer which doesn't cooperate to get in your way.
- mikepurvis 3y ago
  "K8s is not really for stateful systems" As a relative novice in the space, I'm grateful to hear someone say this out loud. K8s seems perfect to me for quickly scaling transient stuff like pipeline workers, web servers, but I've always been pretty leery of giving up the trivial snapshotting and rollbacks and other creature comforts of old-school virtualization when it comes to deploying long running applications, databases, and so on. And I've always felt kind of kind of guilty for not being on board to just mindlessly k8s-all-the-things.
  - snapetom 3y ago
    Our company is finally looking at containerization and orchestration, and one product had the gall to say, Docker isn't for us. The non-technical people gasped because all the other products are moving towards orchestration! Why? it's an ancient Windows Client/Server app. Each "node" manages its own state and communicates with each other in this proprietary, janky-ass way. It takes 10 minutes to start up a node. K8/Swarm isn't going to do squat for this team except maybe launch dev/test environments a little easier.
  - gnufied 3y ago
    >but I've always been pretty leery of giving up the trivial snapshotting and rollbacks and other creature comforts of old-school virtualization when it comes to deploying long running applications, You can do that too with k8s with APIs which support more than just one backend.
- DasIch 3y ago
  Here is an announcement from last year stating that volume expansion is stable since 1.24: https://kubernetes.io/blog/2022/05/05/volume-expansion-ga/ https://kubernetes.io/blog/2022/05/05/volume-expansion-ga/ The zalando postgres-operator also mentions as a feature: > Live volume resize without pod restarts (AWS EBS, PVC)
jauntywundrkind 3y ago
Is the complexity better or worse than alternatives? What are the alternatives? People use complexity as a boogieman to justify throwing together their own really wild chaotic & only so-so tested "simple" alternatives all the time. To me, this feels like a modern wonder. We have layers of responsibility. Many people operate Kubernetes clusters already for all kinds of reasons. It provides a powerful broad base. Now on top of that, we can operate Postgres, with a very smart failover system that's super well tested & broadly used, that leverages this competent starting place. What are the Fears Uncertainties and Doubts you have that make you scared about composition? What would help address specific concerns? To me, this division of responsibilities & use of consistent platform for a variety of needs feels like a huge win. People love "simple" options but they're not. Run naked through the woods like savages option has appeal, but just getting started keeps adding up: Sure, just add some bash scripts for some wal backups that go off-site. Easy! Install pgbouncer like one does, just a quick install, point it at the right systems. Setup some replication. Install and configure more kind of hairy software to make it HA. Configure some TLS cert yourself to not send naked traffic over wire. Add monitoring! Then operationally, how quickly do you think you'll be able to fail over (with ogbouncer staying ok), failback, do an upgrade, add more replicas, lose a replica? Can you rotate cert reliably in a timely fashion? Can your replacement? How well did you document everything? Will you have all the monitoring you need when incidents start coming in, or did you just spitball a couple metrics into place? The alternatives, in my view, are obviously bad. You can do them. Either cheaply with risk or industriously with effort & applied-talent. But having cohesive autonomic systems at our back that try to help, that can faultlessly do many common tasks with perfect accuracy (across unimaginable numbers of systems, with perfect consistency, in record time): that feels like a massively better place in the universe, one that I don't get why so many people kick scream & drag against. Rarely are their arguments well elaborated ("scary"), and their counter-suggestions feel like they massively underrated how multi-faceted & carefully connected production systems are, for good reason, and how hard it can be to remember to not forget to change X when you do Y.
- marcosdumay 3y ago
  > What are the alternatives? Installing the Postgres on the computer, or on a VM. Really, just the fact that some people keep asking this question is enough to question everything else they say. The alternatives are obvious.
znpy 3y ago
> Is there any way to overcome hidden complexity biting your hand other than studying k8s extensively? No. Technology is hard. Operating services is complex, and gets more complex as the scale gets bigger. Any attempt at making stuff simpler is usually either moving complexity elsewhere or making it more expensive (and in some cases, both).
Nican 3y ago
I am happy using CockroachDB. The performance is not as good, since all your database writes require a 2 out of 3 quorum. But managing the database with the CockroachDB is pretty simple, since it can perform a rolling upgrade with no downtime. Upgrades is handled with an operator, and happens by waiting all queries to finish, draining all connections, and restarting the pod with the newer version. The application can connect to any pod without any difference. I perform upgrades twice a year, never really worried about it, and never had any availability problems with the database, even when GCP decides to restart the nodes to update the underlying k8s version.
snapetom 3y ago
I've never been a fan of deploying DBs, either traditional RDBMS or distributed, under containerization/orchestration other than quickly spinning up dev and test environments. Certainly not for production. Databases have been built for high availability for many decades. Sure, maybe RDBMS doesn't scale quite as easily as a node API application, but it's still not really rocket science and you usually have some leeway if you get a spike. Adding Swarm or K8 just seems like redundant and unnecessary complexity.
- steveBK123 3y ago
  Wouldn't most data driven, disk-IO intensive workloads be a bad fit for k8s? Especially when its not purely read-only.. That seems the crux of it.
  - snapetom 3y ago
    Absolutely. I haven't gotten in the weeds of K8 volumes, but certainly have done a lot with Docker volumes including adding some PRs to fix some issues I've ran into. There is a lot under the hood with Docker volumes. With how DBs like to optimize their IO operations, I'm frankly a little amazed how it somehow mostly works between the DB -> orchestrator volumes -> file system.
    - dilyevsky 3y ago
      It works totally fine on k8s with iscsi/fcoe/nvme-backed volumes provided remote targets themselves offer sufficient performance for your db workload
    - p_l 3y ago
      K8s mostly drops docker volumes code (and many other parts of docker, TBH). You can pass a random mount point or even a raw device and it will have only the overhead of kernel IO code for namespaces, overhead you hit anyway. The only "docker graph" stuff is in OCI container data itself.
  - cwiggs 3y ago
mplewis 3y ago
My advice is pay someone to run the cluster, and you can use all that time you saved on k8s complexity to operate the stuff inside the cluster and save some money. Features like k8s Operators are essentially professionally-written code that stands in for a human agent that would take actions such as managing DB nodes, renewing certs, and performing backups. If you use mature operators, you can save a lot of money for a bit of effort.

renegade-otter 3y ago

This kind of resilience is a form of art, and it's also kind of a full-time job. I would not advise trying this for a "side project at work". Generally we all agree that we move to the cloud and it's "fully managed". If it goes down - that's the price we pay. If you have to ask the question "what if the RDS goes down", then you are really in a different universe. That last guarantee of uptime requires a ton of work, testing, and money, because you are all the way up and to the right on the curve of diminishing returns.

candiddevmike 3y ago
> That last guarantee of uptime requires a ton of work, testing, and money, because you are all the way up and to the right on the curve of diminishing returns. Not when it's a core part of your business...?
williamdclt 3y ago
> If you have to ask the question "what if the RDS goes down", then you are really in a different universe. It does go down though, don’t neglect the possibility because it likely will happen. With very average workloads, I’ve seen RDS databases restart unexpectedly, read replicas being completely out of service, and even databases being completely frozen (can’t even connect as root). I’d still go with managed, but it certainly doesn’t give full reliability :) you still have to consider “what if it goes down” - it will!
- debarshri 3y ago
  Problem is that RDS comes at a price. It is purely about operation cost. When you have 1500+ databases these cost add up. At that point, this kind of techniques are required to self host the databases. Price per DB with HPA na VPA is way lower than what you would pay for managed databases as well as you can hire a full time devops+dbadmin and still be cheaper.
  - SlightlyLeftPad 3y ago
    This is a perfect example of a cost savings opportunity that offers no benefit and is really just a foot gun when it comes to operational complexity. The cloud cost savings almost never justify the operational costs here. There’s typically a very long list of other cost optimizations to be made before this would ever be on the table for me. If you’re talking about 1500+ databases and think a single full time devops+dbadmin in some kind of unicorn person is going to be a better option, I feel bad for both of you.
    - p_l 3y ago
      Except the typical cloud prices are dialed for USA, and it's not that unusual in my experience for the cloud savings to be high enough to add more than one highly skilled engineer who will give large positive impact all around.
    - debarshri 3y ago
      I am not implying hiring single full time dbadmin, you can hire a team of them and still be cheaper that what AWS would cost at that scale.
figassis 3y ago
My company also runs on a k8s cluster, but I agree with this fully and I keep my DB outside the cluster. For me, the order of preference is: RDS, Cloud Instance, Bare metal DB server, something else that is not kubernetes. RDS will fail, but it likely will come back up without much action on my part. K8s will fail, you will spend untold human hours figuring out the k8s failure modes, before figuring out the database failure modes (which likely are quite straight forward). It's just a cost that is not worth it.

szszrk 3y ago

Not really. Those are two areas you need to really understand when things go south. In Kubernetes you "solve" a lot of application complexity by abstracting it into things like Helm Charts or Operators. It's fun and easy to use operators to deploy databases, monitoring, mesh, minio... This does not make complexity disappear, it just abstracts it by creating another layer with an interface (k8s API) you may be more familiar with. Whole Kubernetes in practice is adding more and more layers on top of it. All those layers/tools are usually amazing but soon you need some real k8s skills AND some app skills to handle even unavoidable scenarios like ... upgrades. That's why it's so confusing sometimes and has such diverse opinions on. There aren't many environments that run vanilla cluster...

levkk 3y ago

In the case of running Postgres on K8s, the problem arises immediately when you try to resize a data volume and you can't because the API doesn't support it. K8s is not really for stateful systems, yet, and systems like Postgres that prefer to manage their own resources, you don't want another layer which doesn't cooperate to get in your way.

mikepurvis 3y ago
"K8s is not really for stateful systems" As a relative novice in the space, I'm grateful to hear someone say this out loud. K8s seems perfect to me for quickly scaling transient stuff like pipeline workers, web servers, but I've always been pretty leery of giving up the trivial snapshotting and rollbacks and other creature comforts of old-school virtualization when it comes to deploying long running applications, databases, and so on. And I've always felt kind of kind of guilty for not being on board to just mindlessly k8s-all-the-things.
- snapetom 3y ago
  Our company is finally looking at containerization and orchestration, and one product had the gall to say, Docker isn't for us. The non-technical people gasped because all the other products are moving towards orchestration! Why? it's an ancient Windows Client/Server app. Each "node" manages its own state and communicates with each other in this proprietary, janky-ass way. It takes 10 minutes to start up a node. K8/Swarm isn't going to do squat for this team except maybe launch dev/test environments a little easier.
- gnufied 3y ago
  >but I've always been pretty leery of giving up the trivial snapshotting and rollbacks and other creature comforts of old-school virtualization when it comes to deploying long running applications, You can do that too with k8s with APIs which support more than just one backend.
DasIch 3y ago
Here is an announcement from last year stating that volume expansion is stable since 1.24: https://kubernetes.io/blog/2022/05/05/volume-expansion-ga/ https://kubernetes.io/blog/2022/05/05/volume-expansion-ga/ The zalando postgres-operator also mentions as a feature: > Live volume resize without pod restarts (AWS EBS, PVC)

jauntywundrkind 3y ago

Is the complexity better or worse than alternatives? What are the alternatives? People use complexity as a boogieman to justify throwing together their own really wild chaotic & only so-so tested "simple" alternatives all the time. To me, this feels like a modern wonder. We have layers of responsibility. Many people operate Kubernetes clusters already for all kinds of reasons. It provides a powerful broad base. Now on top of that, we can operate Postgres, with a very smart failover system that's super well tested & broadly used, that leverages this competent starting place. What are the Fears Uncertainties and Doubts you have that make you scared about composition? What would help address specific concerns? To me, this division of responsibilities & use of consistent platform for a variety of needs feels like a huge win. People love "simple" options but they're not. Run naked through the woods like savages option has appeal, but just getting started keeps adding up: Sure, just add some bash scripts for some wal backups that go off-site. Easy! Install pgbouncer like one does, just a quick install, point it at the right systems. Setup some replication. Install and configure more kind of hairy software to make it HA. Configure some TLS cert yourself to not send naked traffic over wire. Add monitoring! Then operationally, how quickly do you think you'll be able to fail over (with ogbouncer staying ok), failback, do an upgrade, add more replicas, lose a replica? Can you rotate cert reliably in a timely fashion? Can your replacement? How well did you document everything? Will you have all the monitoring you need when incidents start coming in, or did you just spitball a couple metrics into place? The alternatives, in my view, are obviously bad. You can do them. Either cheaply with risk or industriously with effort & applied-talent. But having cohesive autonomic systems at our back that try to help, that can faultlessly do many common tasks with perfect accuracy (across unimaginable numbers of systems, with perfect consistency, in record time): that feels like a massively better place in the universe, one that I don't get why so many people kick scream & drag against. Rarely are their arguments well elaborated ("scary"), and their counter-suggestions feel like they massively underrated how multi-faceted & carefully connected production systems are, for good reason, and how hard it can be to remember to not forget to change X when you do Y.

marcosdumay 3y ago
> What are the alternatives? Installing the Postgres on the computer, or on a VM. Really, just the fact that some people keep asking this question is enough to question everything else they say. The alternatives are obvious.

znpy 3y ago

> Is there any way to overcome hidden complexity biting your hand other than studying k8s extensively? No. Technology is hard. Operating services is complex, and gets more complex as the scale gets bigger. Any attempt at making stuff simpler is usually either moving complexity elsewhere or making it more expensive (and in some cases, both).

Nican 3y ago

I am happy using CockroachDB. The performance is not as good, since all your database writes require a 2 out of 3 quorum. But managing the database with the CockroachDB is pretty simple, since it can perform a rolling upgrade with no downtime. Upgrades is handled with an operator, and happens by waiting all queries to finish, draining all connections, and restarting the pod with the newer version. The application can connect to any pod without any difference. I perform upgrades twice a year, never really worried about it, and never had any availability problems with the database, even when GCP decides to restart the nodes to update the underlying k8s version.

snapetom 3y ago

I've never been a fan of deploying DBs, either traditional RDBMS or distributed, under containerization/orchestration other than quickly spinning up dev and test environments. Certainly not for production. Databases have been built for high availability for many decades. Sure, maybe RDBMS doesn't scale quite as easily as a node API application, but it's still not really rocket science and you usually have some leeway if you get a spike. Adding Swarm or K8 just seems like redundant and unnecessary complexity.

steveBK123 3y ago
Wouldn't most data driven, disk-IO intensive workloads be a bad fit for k8s? Especially when its not purely read-only.. That seems the crux of it.
- snapetom 3y ago
  Absolutely. I haven't gotten in the weeds of K8 volumes, but certainly have done a lot with Docker volumes including adding some PRs to fix some issues I've ran into. There is a lot under the hood with Docker volumes. With how DBs like to optimize their IO operations, I'm frankly a little amazed how it somehow mostly works between the DB -> orchestrator volumes -> file system.
  - dilyevsky 3y ago
    It works totally fine on k8s with iscsi/fcoe/nvme-backed volumes provided remote targets themselves offer sufficient performance for your db workload
  - p_l 3y ago
    K8s mostly drops docker volumes code (and many other parts of docker, TBH). You can pass a random mount point or even a raw device and it will have only the overhead of kernel IO code for namespaces, overhead you hit anyway. The only "docker graph" stuff is in OCI container data itself.
- cwiggs 3y ago

mplewis 3y ago

My advice is pay someone to run the cluster, and you can use all that time you saved on k8s complexity to operate the stuff inside the cluster and save some money. Features like k8s Operators are essentially professionally-written code that stands in for a human agent that would take actions such as managing DB nodes, renewing certs, and performing backups. If you use mature operators, you can save a lot of money for a bit of effort.