9 ms·

Why would i want to run my database in k8s?

by leetrout 5y ago

voidfunc 5y ago
Why not? Its just a workload scheduler. Some of us have competent ops teams
- longcommonname 5y ago
  What level of competency do you need to run dbs?
  - darthrupert 5y ago
    Poorly? Almost none. Well? Significant.
    - edoceo 5y ago
      Unhelpful and arrogant. We are amateurs in the eyes of the gods.
      - dijit 5y ago
        It’s very easy to set something up for the first time. It’s a bit complex to keep something running for a long time. It’s very difficult to make something fault tolerant with no data loss or corruption. Arrogance is thinking that these things are easy because you can read a quickstart.
      - tinus_hn 5y ago
        It is true though, but it is also true that a ‘poorly’ run database server is good enough for many use cases. There is a reason people are DBAs and they are very well paid.
      - janto 5y ago
        Arrogant and foolish to declare no-one can be high skilled.
      - darthrupert 5y ago
        You're right. But so am I.
- ryanisnan 5y ago
  I think while true, this is an incredibly reductive take on what Kubernetes really is.
manigandham 5y ago
Because kubernetes is a fantastic platform that provides a lot of node and app management functionality out of the box. Stateful services like databases are not really a challenge anymore given recent developments in both K8S and database containerization.
- tiew9Vii 5y ago
  With a database I generally want to have it on its own dedicated host with fast NVME drives, memory, cpu etc, maybe some OS specific system level configuration options for performance etc. I don't want to binpack it on a cluster of nodes each contending for system resources. Also RDBMS databases are not really designed to be ephemeral/elastic, they are stateful/persistent by nature. What does Kubernetes offer in this scenario? I'm still hesitant to run things like DB's on K8's but have absolutely no problems running cloud native apps on K8's which can run ephemerally, be bin packed and only needing compute as all state is external to where the app is being executed.
  - alphabettsy 5y ago
    StatefulSets, Node Groups, and Node Affinity/Taints solve some of those issues.
    - karolist 5y ago
      A database is not a service well suited for ephemeral, quick scale down and scale up container workloads that K8s really excells at, and if you have nodes specifically configured to run databases and nothing else then you should simply run them there directly and not mangle K8s to replicate what will essentially be equivalent to running them there directly.
      - manigandham 5y ago
        K8S runs workloads across servers in a declarative fashion. Ephemeral or stateful doesn't make a difference anymore. Whatever you use to manage the nodes directly is basically what K8S provides (and more) already, so you're really just replicating effort and complexity instead.
evil-olive 5y ago
if you run your stateless services on an orchestration system like k8s, you have two options: a) run your stateful services (such as Postgres) on the same orchestration system b) choose a second orchestration system (often a more "traditional" infrastructure provisioning system like Chef/Puppet/Ansible/SaltStack) that will only manage your stateful services, and deal with the care & feeding & mental overhead of having two things that do fairly similar jobs and need to interact with each other (because of course your k8s-hosted stateless services need to be able to look up Postgres in DNS) (option C would be to use a managed service like RDS...but options A and B assume you've already made a decision, for whatever reason, to run the stateful services yourself)
- morelisp 5y ago
  If you choose a) you don't really get to avoid b), since something still has to bootstrap your Kubernetes nodes themselves. So it's more a question whether you'd rather write a Kubernetes manifest or a Salt/Ansible config; and with the complexity and limitations of a lot of storage operators, and no real "orchestration" benefit once you have pinned deployments to particular nodes anyway, sometimes it's the latter. The main force that's pushing us to move our storage into K8s isn't simplifying our node configuration, but to keep parity between our prod and more dynamic staging environments. For the latter k8s storage operators are great because we don't generally care about data durability or performance.
  - cdogl 5y ago
    > If you choose a) you don't really get to avoid b), since something still has to bootstrap your Kubernetes nodes themselves Not quite true if you use a managed K8S offering - and it seems a self-managed cluster is something to be avoided until absolutely necessary. Even with a managed cluster like GKE there's plenty of operational overhead to keep everything up-to-date and working properly.
    - morelisp 5y ago
      If you're running on a managed K8s offering yet not using the managed storage, you're getting ripped off. (I'm well aware this describes a lot of companies / projects.)
dagw 5y ago
At least in my case, because your ops team has decided that everything that gets deployed, gets deployed in k8s (unless you have a Very! good reason not to)
- dijit 5y ago
  Kubernetes is great but not everything fits in kubernetes, this is a bad ops team if this is the case.
  - jpgvm 5y ago
    Note the "except if you have a very good reason" caveat. I think this is a good ops team mantra. PostgreSQL unless you have a very good reason otherwise, and k8s unless you have a very good reason otherwise. Decisions like this allow for economies of scale with tooling, monitoring/observability, etc. Saying no to snowflakes is an important part of having a good ops team.
    - dijit 5y ago
      Guess it depends on who is defining “very”. A persistent connection is antithetical to the design of kubernetes. Persistent storage is hard to get right in kubernetes and puts a lot of pressure on the cluster being perfect, which is something a good ops team will tell you is hard. If your application is stateless http services then kubernetes is great, but there are common things that don’t fit well, and forcing people into a solution which may require a complete rethinking is not very pragmatic.
  - dagw 5y ago
    Eh, for what it's worth they've deployed a couple of PostgreSQL instances for me using Kubernetes and it's worked exactly like I've wanted it to. But then again I've never had push those instances to breaking point either when it comes to IO or concurrency so who knows.
- tveita 5y ago
  A middle ground would be provisioning managed services through Kubernetes resources using a controller like Config Connector (GCP) or ACK (AWS) https://cloud.google.com/config-connector/docs/reference/resource-docs/sql/sqlinstance https://cloud.google.com/config-connector/docs/reference/res... https://aws-controllers-k8s.github.io/community/reference/rds/v1alpha1/dbinstance/ https://aws-controllers-k8s.github.io/community/reference/rd...
hijinks 5y ago
20 years ago people said the same thing about running DBs on virtual machines. Now almost all databases are run on VMs
- mbreese 5y ago
  But that’s really not a great comparison, is it? Running on a VM is much more like running on a plain physical node. Running on K8s implies a whole new level of management overhead, that just isn’t there on a single node system. In this case, you should be asking “will we all be running databases in clustered mode in the future?” I think that answer still up for debate.
  - jatone 5y ago
    containers are actually a really good fit for dbs. they do one thing and one thing only and for a really really long time without many changes. and the answer is we will definitely be running DBs in clustered mode in the future. its not really a debate its an eventuality. without clusters you lose on reliability and maintenance QOL improvements. without clusters its very hard to upgrade systems effectively leaving systems to stagnate and decay both in performance (Hard/Software improvements aka kernel updates or inbternal DB improvements) and reliability.
    - morelisp 5y ago
      The question isn't whether DBs will be running in clusters - they all pretty much do, or can - but will those clusters be orchestrated by something first designed for dynamic scaling of stateless services, treating all nodes as varying-sized pools of otherwise mostly-homogeneous hyperconverged compute/storage? Or, how much are you willing to pay in $/performance/dev time to force them into such infrastructure? Or will they be on something more purpose-built, either designed from the start to handle storage more flexibly at a higher level? Or conversely something more specific and lower-level to extract maximum performance for their particular storage patterns?
  - gizdan 5y ago
    > Running on a VM is much more like running on a plain physical node. Sure, and containers are just plain processes on a VM. The exact code that controls processes in a VM, also controls processes inside a container. As long as the underlying data volume is reliable, it really doesn't matter whether you're running it outside of a container or not. Volumes for K8s have a come a long way since its early days.
rafaelturk 5y ago
We run all our databases in K8S, approx 80 DBs. We're a fintech that requires HA so all databases run with at least 3 notes. Maging DB clusters with K8S only requires 1 file. If we need to increase storage this can be achieved in minutes, if we need to take one node down we basically can do it anytime during the day.. Every now and then we discuss moving DBs to AWS RDS (or equivalent) however for our transaction volume RDS costs are prohibitive.
- cbushko 5y ago
  What DB are you running?
- dijit 5y ago
  I just checked your comment history and it appears that you like to troll. How can I know that this isn't one of those cases? I know some folks in fintech and while obviously they're not _you_, they're definitely not the sort to be wishy washy with data, if anything they're quite conservative when it comes to how the databases are managed. Any incidental complexity is incredibly discouraged.

Why would i want to run my database in k8s?

voidfunc 5y ago

Why not? Its just a workload scheduler. Some of us have competent ops teams

longcommonname 5y ago
What level of competency do you need to run dbs?
- darthrupert 5y ago
  Poorly? Almost none. Well? Significant.
  - edoceo 5y ago
    Unhelpful and arrogant. We are amateurs in the eyes of the gods.
    - dijit 5y ago
      It’s very easy to set something up for the first time. It’s a bit complex to keep something running for a long time. It’s very difficult to make something fault tolerant with no data loss or corruption. Arrogance is thinking that these things are easy because you can read a quickstart.
    - tinus_hn 5y ago
      It is true though, but it is also true that a ‘poorly’ run database server is good enough for many use cases. There is a reason people are DBAs and they are very well paid.
    - janto 5y ago
      Arrogant and foolish to declare no-one can be high skilled.
    - darthrupert 5y ago
      You're right. But so am I.
ryanisnan 5y ago
I think while true, this is an incredibly reductive take on what Kubernetes really is.

manigandham 5y ago

Because kubernetes is a fantastic platform that provides a lot of node and app management functionality out of the box. Stateful services like databases are not really a challenge anymore given recent developments in both K8S and database containerization.

tiew9Vii 5y ago
With a database I generally want to have it on its own dedicated host with fast NVME drives, memory, cpu etc, maybe some OS specific system level configuration options for performance etc. I don't want to binpack it on a cluster of nodes each contending for system resources. Also RDBMS databases are not really designed to be ephemeral/elastic, they are stateful/persistent by nature. What does Kubernetes offer in this scenario? I'm still hesitant to run things like DB's on K8's but have absolutely no problems running cloud native apps on K8's which can run ephemerally, be bin packed and only needing compute as all state is external to where the app is being executed.
- alphabettsy 5y ago
  StatefulSets, Node Groups, and Node Affinity/Taints solve some of those issues.
  - karolist 5y ago
    A database is not a service well suited for ephemeral, quick scale down and scale up container workloads that K8s really excells at, and if you have nodes specifically configured to run databases and nothing else then you should simply run them there directly and not mangle K8s to replicate what will essentially be equivalent to running them there directly.
    - manigandham 5y ago
      K8S runs workloads across servers in a declarative fashion. Ephemeral or stateful doesn't make a difference anymore. Whatever you use to manage the nodes directly is basically what K8S provides (and more) already, so you're really just replicating effort and complexity instead.

evil-olive 5y ago

if you run your stateless services on an orchestration system like k8s, you have two options: a) run your stateful services (such as Postgres) on the same orchestration system b) choose a second orchestration system (often a more "traditional" infrastructure provisioning system like Chef/Puppet/Ansible/SaltStack) that will only manage your stateful services, and deal with the care & feeding & mental overhead of having two things that do fairly similar jobs and need to interact with each other (because of course your k8s-hosted stateless services need to be able to look up Postgres in DNS) (option C would be to use a managed service like RDS...but options A and B assume you've already made a decision, for whatever reason, to run the stateful services yourself)

morelisp 5y ago
If you choose a) you don't really get to avoid b), since something still has to bootstrap your Kubernetes nodes themselves. So it's more a question whether you'd rather write a Kubernetes manifest or a Salt/Ansible config; and with the complexity and limitations of a lot of storage operators, and no real "orchestration" benefit once you have pinned deployments to particular nodes anyway, sometimes it's the latter. The main force that's pushing us to move our storage into K8s isn't simplifying our node configuration, but to keep parity between our prod and more dynamic staging environments. For the latter k8s storage operators are great because we don't generally care about data durability or performance.
- cdogl 5y ago
  > If you choose a) you don't really get to avoid b), since something still has to bootstrap your Kubernetes nodes themselves Not quite true if you use a managed K8S offering - and it seems a self-managed cluster is something to be avoided until absolutely necessary. Even with a managed cluster like GKE there's plenty of operational overhead to keep everything up-to-date and working properly.
  - morelisp 5y ago
    If you're running on a managed K8s offering yet not using the managed storage, you're getting ripped off. (I'm well aware this describes a lot of companies / projects.)

dagw 5y ago

At least in my case, because your ops team has decided that everything that gets deployed, gets deployed in k8s (unless you have a Very! good reason not to)

dijit 5y ago
Kubernetes is great but not everything fits in kubernetes, this is a bad ops team if this is the case.
- jpgvm 5y ago
  Note the "except if you have a very good reason" caveat. I think this is a good ops team mantra. PostgreSQL unless you have a very good reason otherwise, and k8s unless you have a very good reason otherwise. Decisions like this allow for economies of scale with tooling, monitoring/observability, etc. Saying no to snowflakes is an important part of having a good ops team.
  - dijit 5y ago
    Guess it depends on who is defining “very”. A persistent connection is antithetical to the design of kubernetes. Persistent storage is hard to get right in kubernetes and puts a lot of pressure on the cluster being perfect, which is something a good ops team will tell you is hard. If your application is stateless http services then kubernetes is great, but there are common things that don’t fit well, and forcing people into a solution which may require a complete rethinking is not very pragmatic.
- dagw 5y ago
  Eh, for what it's worth they've deployed a couple of PostgreSQL instances for me using Kubernetes and it's worked exactly like I've wanted it to. But then again I've never had push those instances to breaking point either when it comes to IO or concurrency so who knows.
tveita 5y ago
A middle ground would be provisioning managed services through Kubernetes resources using a controller like Config Connector (GCP) or ACK (AWS) https://cloud.google.com/config-connector/docs/reference/resource-docs/sql/sqlinstance https://cloud.google.com/config-connector/docs/reference/res... https://aws-controllers-k8s.github.io/community/reference/rds/v1alpha1/dbinstance/ https://aws-controllers-k8s.github.io/community/reference/rd...

hijinks 5y ago

20 years ago people said the same thing about running DBs on virtual machines. Now almost all databases are run on VMs

mbreese 5y ago
But that’s really not a great comparison, is it? Running on a VM is much more like running on a plain physical node. Running on K8s implies a whole new level of management overhead, that just isn’t there on a single node system. In this case, you should be asking “will we all be running databases in clustered mode in the future?” I think that answer still up for debate.
- jatone 5y ago
  containers are actually a really good fit for dbs. they do one thing and one thing only and for a really really long time without many changes. and the answer is we will definitely be running DBs in clustered mode in the future. its not really a debate its an eventuality. without clusters you lose on reliability and maintenance QOL improvements. without clusters its very hard to upgrade systems effectively leaving systems to stagnate and decay both in performance (Hard/Software improvements aka kernel updates or inbternal DB improvements) and reliability.
  - morelisp 5y ago
    The question isn't whether DBs will be running in clusters - they all pretty much do, or can - but will those clusters be orchestrated by something first designed for dynamic scaling of stateless services, treating all nodes as varying-sized pools of otherwise mostly-homogeneous hyperconverged compute/storage? Or, how much are you willing to pay in $/performance/dev time to force them into such infrastructure? Or will they be on something more purpose-built, either designed from the start to handle storage more flexibly at a higher level? Or conversely something more specific and lower-level to extract maximum performance for their particular storage patterns?
- gizdan 5y ago
  > Running on a VM is much more like running on a plain physical node. Sure, and containers are just plain processes on a VM. The exact code that controls processes in a VM, also controls processes inside a container. As long as the underlying data volume is reliable, it really doesn't matter whether you're running it outside of a container or not. Volumes for K8s have a come a long way since its early days.

rafaelturk 5y ago

We run all our databases in K8S, approx 80 DBs. We're a fintech that requires HA so all databases run with at least 3 notes. Maging DB clusters with K8S only requires 1 file. If we need to increase storage this can be achieved in minutes, if we need to take one node down we basically can do it anytime during the day.. Every now and then we discuss moving DBs to AWS RDS (or equivalent) however for our transaction volume RDS costs are prohibitive.

cbushko 5y ago
What DB are you running?
dijit 5y ago
I just checked your comment history and it appears that you like to troll. How can I know that this isn't one of those cases? I know some folks in fintech and while obviously they're not _you_, they're definitely not the sort to be wishy washy with data, if anything they're quite conservative when it comes to how the databases are managed. Any incidental complexity is incredibly discouraged.