4 ms·

There is no good reason to run non-test database workloads in Kubernetes or Docker. Databases are designed to sit close to the hardware and have a stable, dedic

There is no good reason to run non-test database workloads in Kubernetes or Docker. Databases are designed to sit close to the hardware and have a stable, dedicated chunk of resources for a long time, whereas Kubernetes pods are subject to vaporization at any moment. Databases traditionally have fought the operating system to try and maintain enough control to remain performant. Introducing additional layers into this would be dubious at the best of times, but when it's something fundamentally contrary to the application's nature like stateless orchestration, it's pure farce. There could not be an application worse-suited to running in Kubernetes et al than a traditional database. Anyone claiming something that rams this square peg into that round hole is "production ready" is showing that they're an empty husk and shouldn't be trusted near anything important. Note the downvotes already rolling in less than two minutes after I posted this. This subject is a major third rail here. It goes against the agenda of very powerful people and my account has been censured in the past specifically for making this particular argument, that database workloads and Kubernetes don't mix. Keep that in mind when you're asking HN for their experience on this (or any other topic that YC considers critical to the interest of their investments -- they've shown that they're willing to taint the discussion if it gets too dicey).

ownagefool 8y ago
I assume you also consider databases on cloud with mounted block storage as not production ready too?
- cookiecaper 8y ago
  For the record, I have to use my reply allocation sparingly, since usually when I start talking about this I'm mysteriously throttled for long periods. That said -- no, that's not the same thing at all. Barring anomalous conditions, VMs run as long as you keep them running. They won't be reaped and rescheduled onto some other node in the cluster, whether by automated rebalancing processes or by manual `kubectl delete po...` or `kubectl drain`. You can easily set up a VM that will behave more-or-less like conventional hardware if we ignore the perf hit. This is a pretty simple thing. The reason people say you need to make your apps "12 factor" when you go to k8s is because it doesn't work well if your app cares about state. Databases care deeply about state. You can't just kill a DB server and spin up a new one to pick up where it left off. You can't parallelize a DB workload by spinning up 8 little DB nodes. It's not a web server and it just doesn't work like that. Things like CockroachDB exist specifically because normal databases don't work like that. This is where people usually bring up things like annotations, labels, StatefulSets, etc. First, note that the facilities that accommodate stateful workloads are not priorities for Kubernetes and are generally not well-tested or consistent. This wouldn't be a news story or an independent project if they were. Second, please realize you're doing all of that work to try and make Kubernetes do something it's not really designed to do, with potential negative impact on the availability and scheduling processes for the applications that do work well on Kubernetes, when you could just spin a VM and avoid all of these issues entirely. There's no reason to put a production DB on k8s other than cargo culting.
  - DasIch 8y ago
    EC2 instances do go down, EBS volumes fail, hardware fails. Maybe not as frequently as pods in Kubernetes get evicted but at sufficient scale it does occur frequently enough overall that you do need to find ways to handle this automatically without human intervention. Once you’ve achieved that whether your database runs on a VM or in Kubernetes doesn’t make a difference really. Granted, if your not at that scale, running a database in Kubernetes is probably not the best of ideas. That has nothing to do with Kubernetes though, that’s because running a stateful service with decent working backup, recovery and automated failover is difficult in any case. If that’s not your job, you’re probably better off using RDS or something equivalent.
  - cpuguy83 8y ago
    A database is an application like any other. Containers are about managing the lifecycle of the process and container managers assist in getting the right state to a container. Wether or not a container has state or not doesn't make it easier or harder to run in a container. If you aren't managing your state, then yeah you will run into a nightmare when trying to containerize stateful apps... or running them at all. You will literally have the same problems with a VM or physical hardware. It's important to separate state management from process management. A stateful application is absolutely not harder to contaknerize than a stateless one. Rather it is simply just harder to run stateful applications in any regard. I would personally argue that it is easier to run a stateful app with a container manager. I know it sounds crazy but... keep in mind container tools are cenetered around what each individual application requires and the tooling tends to make it easier to express and assist in managing the state requirements of that application. For that matter you can even prevent the scheduler from scheduling your stateful app on a new node, which seems to be the answer for the crux of the argument against containerizing a stateful app.
    - wbl 8y ago
      Module the development cycle of push and run a new container being incompatible with state. Sure google containerizes everything but they invested effort.
- user5994461 8y ago
  EBS support in Kubernetes was considered experimental as of last year. Not sure about now. I recall a few nasty issues in the GitHub with data loss or unmountable volumes for the early adopters, with the official answer along the lines of "implementation is in progress".
  - gnufied 8y ago
    I do not think EBS support in Kubernetes was experimental in 2017. I am one of the maintainers of in-tree EBS driver and we have tried our best to iron out any bugs reported. There are still bugs, I do not disagree. Data loss bugs are considered top priority and I am not aware of any open such bugs against EBS driver.
    - user5994461 8y ago
      Pretty sure the bugs in question were closed. You'll excuse me but no time to go through the history and dig up the tickets for reference.
markbnj 8y ago
The same good reasons for running any workload on k8s apply to databases as well, it's just that they are more complicated than stateless services due to the (no surprise) state and the clustering/control protocol things that often accompany HA data stores. Kubernetes has the tools available to manage state now, and many (maybe most) databases now offer some support for its native discovery model. So all in all my current preferred strategy for databases is to prefer hosted if its available (cloudsql, elastic db), k8s if it isn't, and vms if it won't work on k8s.
eloff 8y ago
I downvoted you because of that whole conspiracy theory you tacked onto the end of your post. But I fully agree that kubernetes and containers are not well suited to running production databases. In theory they could achieve parity with a dedicated machine or VM, but they're still a long ways from that - and it makes it very easy to lose your data. I was recovering a database where the persistent volume wasn't setup right and the container got killed and restarted. It was just before the holidays and it was a nightmare because everyone was on vacation. Yeah you could get into that kind of problem with a VM or dedicated machine, but the bar is a lot higher, you'd need some kind of hardware failure. Kubernetes makes it really easy to shoot yourself in the foot when running databases.
- alexandre_m 8y ago
  "I was recovering a database where the persistent volume wasn't setup right and the container got killed and restarted." In other words, your database application was using scratch storage instead of persistent volumes? What this anecdote shows is that the developers or admins responsible to setup the database didn't do it properly. Also, testing failures and data recovery should be your priority before going to production. I don't see how you could blame that on software.
  - eloff 8y ago
    Not 100% sure that it was using scratch, but something went wrong with the persistence. The point is not to say it wasn't human error - clearly it was, but it's an error that wouldn't have been as easy to make without kubernetes. There's a cost to running a database on k8s that largely people ignore. That's before you start talking about backups and recovery which also get harder and require more manual work with more potential for error.
derefr 8y ago
Using Kubernetes doesn't imply using Docker, even. K8s is 99% an orchestration system, like Terraform or CloudFormation. One resource among many that it orchestrates is containers. It can also orchestrate regular VMs. That being said, I also disagree that Docker isn't suited to running a DBMS, assuming you actually have a large enterprise (or cloud) datacenter backing your Docker daemon. In such cases: • You'll probably have a large enough pool of Docker machines (k8s or not) that you're going to be deploying your DBMS container in a way that reserves an entire instance just for it (or it + its accessory containers); • You'll probably have a SAN, and you'll have many enterprise-y reasons (e.g. live VM migration) to prefer backing your DBMS with said SAN, rather than with local instance storage. If both of those are true, then Docker has no disadvantages compared to deploying your DBMS as a raw VM.
- peterwwillis 8y ago
  As an "enterprisey" person, I disagree. I've seen a lot of enterprise infrastructure that looks like toddlers built it out of lincoln logs. And I've seen SANs lose connectivity much more often than a pool of independent local disks all going bad at once. On top of that, databases run on VMs that aren't on hypervisors dedicated for running databases results in shitty adminning and overcrowded VM pools destroying database performance+reliability. Cloud-ish infrastructure is often good for running distributed decentralized databases, but try running Oracle in a bunch of Docker containers on a crappy OpenStack cluster and soon you'll be crying into your scotch. These efforts to make people think it's a good idea to run databases on K8s are misleading people, and god help those poor teams that waste years trying to stabilize something that a fancy web page and a youtube tutorial said was a great idea.

ownagefool 8y ago

I assume you also consider databases on cloud with mounted block storage as not production ready too?

cookiecaper 8y ago
For the record, I have to use my reply allocation sparingly, since usually when I start talking about this I'm mysteriously throttled for long periods. That said -- no, that's not the same thing at all. Barring anomalous conditions, VMs run as long as you keep them running. They won't be reaped and rescheduled onto some other node in the cluster, whether by automated rebalancing processes or by manual `kubectl delete po...` or `kubectl drain`. You can easily set up a VM that will behave more-or-less like conventional hardware if we ignore the perf hit. This is a pretty simple thing. The reason people say you need to make your apps "12 factor" when you go to k8s is because it doesn't work well if your app cares about state. Databases care deeply about state. You can't just kill a DB server and spin up a new one to pick up where it left off. You can't parallelize a DB workload by spinning up 8 little DB nodes. It's not a web server and it just doesn't work like that. Things like CockroachDB exist specifically because normal databases don't work like that. This is where people usually bring up things like annotations, labels, StatefulSets, etc. First, note that the facilities that accommodate stateful workloads are not priorities for Kubernetes and are generally not well-tested or consistent. This wouldn't be a news story or an independent project if they were. Second, please realize you're doing all of that work to try and make Kubernetes do something it's not really designed to do, with potential negative impact on the availability and scheduling processes for the applications that do work well on Kubernetes, when you could just spin a VM and avoid all of these issues entirely. There's no reason to put a production DB on k8s other than cargo culting.
- DasIch 8y ago
  EC2 instances do go down, EBS volumes fail, hardware fails. Maybe not as frequently as pods in Kubernetes get evicted but at sufficient scale it does occur frequently enough overall that you do need to find ways to handle this automatically without human intervention. Once you’ve achieved that whether your database runs on a VM or in Kubernetes doesn’t make a difference really. Granted, if your not at that scale, running a database in Kubernetes is probably not the best of ideas. That has nothing to do with Kubernetes though, that’s because running a stateful service with decent working backup, recovery and automated failover is difficult in any case. If that’s not your job, you’re probably better off using RDS or something equivalent.
- cpuguy83 8y ago
  A database is an application like any other. Containers are about managing the lifecycle of the process and container managers assist in getting the right state to a container. Wether or not a container has state or not doesn't make it easier or harder to run in a container. If you aren't managing your state, then yeah you will run into a nightmare when trying to containerize stateful apps... or running them at all. You will literally have the same problems with a VM or physical hardware. It's important to separate state management from process management. A stateful application is absolutely not harder to contaknerize than a stateless one. Rather it is simply just harder to run stateful applications in any regard. I would personally argue that it is easier to run a stateful app with a container manager. I know it sounds crazy but... keep in mind container tools are cenetered around what each individual application requires and the tooling tends to make it easier to express and assist in managing the state requirements of that application. For that matter you can even prevent the scheduler from scheduling your stateful app on a new node, which seems to be the answer for the crux of the argument against containerizing a stateful app.
  - wbl 8y ago
    Module the development cycle of push and run a new container being incompatible with state. Sure google containerizes everything but they invested effort.
user5994461 8y ago
EBS support in Kubernetes was considered experimental as of last year. Not sure about now. I recall a few nasty issues in the GitHub with data loss or unmountable volumes for the early adopters, with the official answer along the lines of "implementation is in progress".
- gnufied 8y ago
  I do not think EBS support in Kubernetes was experimental in 2017. I am one of the maintainers of in-tree EBS driver and we have tried our best to iron out any bugs reported. There are still bugs, I do not disagree. Data loss bugs are considered top priority and I am not aware of any open such bugs against EBS driver.
  - user5994461 8y ago
    Pretty sure the bugs in question were closed. You'll excuse me but no time to go through the history and dig up the tickets for reference.

markbnj 8y ago

The same good reasons for running any workload on k8s apply to databases as well, it's just that they are more complicated than stateless services due to the (no surprise) state and the clustering/control protocol things that often accompany HA data stores. Kubernetes has the tools available to manage state now, and many (maybe most) databases now offer some support for its native discovery model. So all in all my current preferred strategy for databases is to prefer hosted if its available (cloudsql, elastic db), k8s if it isn't, and vms if it won't work on k8s.

eloff 8y ago

I downvoted you because of that whole conspiracy theory you tacked onto the end of your post. But I fully agree that kubernetes and containers are not well suited to running production databases. In theory they could achieve parity with a dedicated machine or VM, but they're still a long ways from that - and it makes it very easy to lose your data. I was recovering a database where the persistent volume wasn't setup right and the container got killed and restarted. It was just before the holidays and it was a nightmare because everyone was on vacation. Yeah you could get into that kind of problem with a VM or dedicated machine, but the bar is a lot higher, you'd need some kind of hardware failure. Kubernetes makes it really easy to shoot yourself in the foot when running databases.

alexandre_m 8y ago
"I was recovering a database where the persistent volume wasn't setup right and the container got killed and restarted." In other words, your database application was using scratch storage instead of persistent volumes? What this anecdote shows is that the developers or admins responsible to setup the database didn't do it properly. Also, testing failures and data recovery should be your priority before going to production. I don't see how you could blame that on software.
- eloff 8y ago
  Not 100% sure that it was using scratch, but something went wrong with the persistence. The point is not to say it wasn't human error - clearly it was, but it's an error that wouldn't have been as easy to make without kubernetes. There's a cost to running a database on k8s that largely people ignore. That's before you start talking about backups and recovery which also get harder and require more manual work with more potential for error.

derefr 8y ago

Using Kubernetes doesn't imply using Docker, even. K8s is 99% an orchestration system, like Terraform or CloudFormation. One resource among many that it orchestrates is containers. It can also orchestrate regular VMs. That being said, I also disagree that Docker isn't suited to running a DBMS, assuming you actually have a large enterprise (or cloud) datacenter backing your Docker daemon. In such cases: • You'll probably have a large enough pool of Docker machines (k8s or not) that you're going to be deploying your DBMS container in a way that reserves an entire instance just for it (or it + its accessory containers); • You'll probably have a SAN, and you'll have many enterprise-y reasons (e.g. live VM migration) to prefer backing your DBMS with said SAN, rather than with local instance storage. If both of those are true, then Docker has no disadvantages compared to deploying your DBMS as a raw VM.

peterwwillis 8y ago
As an "enterprisey" person, I disagree. I've seen a lot of enterprise infrastructure that looks like toddlers built it out of lincoln logs. And I've seen SANs lose connectivity much more often than a pool of independent local disks all going bad at once. On top of that, databases run on VMs that aren't on hypervisors dedicated for running databases results in shitty adminning and overcrowded VM pools destroying database performance+reliability. Cloud-ish infrastructure is often good for running distributed decentralized databases, but try running Oracle in a bunch of Docker containers on a crappy OpenStack cluster and soon you'll be crying into your scotch. These efforts to make people think it's a good idea to run databases on K8s are misleading people, and god help those poor teams that waste years trying to stabilize something that a fancy web page and a youtube tutorial said was a great idea.