12 ms·

While I have completely embraced running stateless services in Docker, I have been hesitant to migrate the database layer to containers. While I have not teste

While I have completely embraced running stateless services in Docker, I have been hesitant to migrate the database layer to containers. While I have not tested it personally, I have seen numerous reports of performance issues when using volumes. Is this no longer an issue, or was it limited to bind mounts? Do volumes not use the storage driver? Also, I have run into permission issues when using volumes with Docker, which I'm sure was just my own ignorance but it does seem like a cause for confusion and potential error. I have read through the documentation on the linked page, and the quickstart guides for KubeDB seems great for getting up and running, but I do worry about situations like if an automated PG database failover can't reconcile a timeline, there isn't much documentation on failover at all and this could add significant complexity to something that is already a potential nightmare. Anyone care to share their experiences running production databases in k8s?

cookiecaper 8y ago
There is no good reason to run non-test database workloads in Kubernetes or Docker. Databases are designed to sit close to the hardware and have a stable, dedicated chunk of resources for a long time, whereas Kubernetes pods are subject to vaporization at any moment. Databases traditionally have fought the operating system to try and maintain enough control to remain performant. Introducing additional layers into this would be dubious at the best of times, but when it's something fundamentally contrary to the application's nature like stateless orchestration, it's pure farce. There could not be an application worse-suited to running in Kubernetes et al than a traditional database. Anyone claiming something that rams this square peg into that round hole is "production ready" is showing that they're an empty husk and shouldn't be trusted near anything important. Note the downvotes already rolling in less than two minutes after I posted this. This subject is a major third rail here. It goes against the agenda of very powerful people and my account has been censured in the past specifically for making this particular argument, that database workloads and Kubernetes don't mix. Keep that in mind when you're asking HN for their experience on this (or any other topic that YC considers critical to the interest of their investments -- they've shown that they're willing to taint the discussion if it gets too dicey).
- ownagefool 8y ago
  I assume you also consider databases on cloud with mounted block storage as not production ready too?
  - cookiecaper 8y ago
    For the record, I have to use my reply allocation sparingly, since usually when I start talking about this I'm mysteriously throttled for long periods. That said -- no, that's not the same thing at all. Barring anomalous conditions, VMs run as long as you keep them running. They won't be reaped and rescheduled onto some other node in the cluster, whether by automated rebalancing processes or by manual `kubectl delete po...` or `kubectl drain`. You can easily set up a VM that will behave more-or-less like conventional hardware if we ignore the perf hit. This is a pretty simple thing. The reason people say you need to make your apps "12 factor" when you go to k8s is because it doesn't work well if your app cares about state. Databases care deeply about state. You can't just kill a DB server and spin up a new one to pick up where it left off. You can't parallelize a DB workload by spinning up 8 little DB nodes. It's not a web server and it just doesn't work like that. Things like CockroachDB exist specifically because normal databases don't work like that. This is where people usually bring up things like annotations, labels, StatefulSets, etc. First, note that the facilities that accommodate stateful workloads are not priorities for Kubernetes and are generally not well-tested or consistent. This wouldn't be a news story or an independent project if they were. Second, please realize you're doing all of that work to try and make Kubernetes do something it's not really designed to do, with potential negative impact on the availability and scheduling processes for the applications that do work well on Kubernetes, when you could just spin a VM and avoid all of these issues entirely. There's no reason to put a production DB on k8s other than cargo culting.
    - DasIch 8y ago
      EC2 instances do go down, EBS volumes fail, hardware fails. Maybe not as frequently as pods in Kubernetes get evicted but at sufficient scale it does occur frequently enough overall that you do need to find ways to handle this automatically without human intervention. Once you’ve achieved that whether your database runs on a VM or in Kubernetes doesn’t make a difference really. Granted, if your not at that scale, running a database in Kubernetes is probably not the best of ideas. That has nothing to do with Kubernetes though, that’s because running a stateful service with decent working backup, recovery and automated failover is difficult in any case. If that’s not your job, you’re probably better off using RDS or something equivalent.
ofrzeta 8y ago
Local volumes don't use the storage driver. Therefore there should be no performance penalty as far as I know. EDIT: There's probably some overhead with the implementation of namespaces and cgroups but I have not found any reliable sources about the quantities. As a side not if you are using memory limits you will have a performance penalty: "Memory and swap accounting incur an overhead of about 1% of the total available memory and a 10% overall performance degradation, even if Docker is not running." (from the Docker docs). This will probably affect databases running in containers as well.
- gnufied 8y ago
  Yeah this can't be stressed enough. Persistent volumes in Kubernetes are simply bind mounts that exist in your container's namespace. They don't go through docker's storage driver at all. There should be zero penalty for using persistent volumes this way within containers.
  - cpuguy83 8y ago
    Sorry, this seems to insinuate that Docker is slowing down (or using slow) storage... this is completely false. Volumes in docker are just host bind mounts. Now depending on your driver/opts (similar to K8s PV backends) this storage can come from anywhere and performance of the volume is totally dependent on the type of storage being used. Now, the container fs that docker sets up is (usually) using a CoW filesystem, and as there is overhead there... but volumes are specifically designed to bypass the container fs.
    - gnufied 8y ago
      Hmm, sorry I did not mean to imply that. It is indeed true that both docker volumes and k8s PVs are just host bind mounts. However I did mean to clarify that - k8s does not uses docker volumes. Whatever performance issues one notices could either be associated with storage provider or container writable layer.
user5994461 8y ago
Running production databases in docker last year: https://thehftguy.com/2016/11/01/docker-in-production-an-history-of-failure/ https://thehftguy.com/2016/11/01/docker-in-production-an-his... Performance issues should be the least of your concern. The docker deamon and container simply hanged because of filesystem issues on CentOS 6. I worked at a company that was dockerizing their stateless services, then planning to dockerize their cassandra databases. Multiple contractors involved. Stateless services failed periodically because of the above issue. Load balancers can failover automatically, broken nodes are rebooted from time to time, limited impact. Noone cared, just a daily deployment routine. I fear the day the cassandra dockerization would happen. They'd lose their entire customer data (hundreds of millions of customers) once two nodes would fail simultaneously, which happened a lot on the stateless services. Thankfully the project never started and the company didn't go bankrupt. Pretty sure employees moved around and plans got canceled. Expect a lot of instability in docker around filesystem, performance issues and race conditions. Low volume stateless web servers don't get to trigger issues much, but databases do.
- gnufied 8y ago
  I can't possibly hope to change your mind but stability issues with union filesystem driver in docker(part of it was not even docker's problem) and persistent volumes of kubernetes are two very different things. Cassandra running standalone on host(and crashing) is no different from cassandra crashing when running using a PV inside a container. Moreover - all/most Linux distros have switched to using Overlay2 as default driver. If you are running latest version of RHEL/CentOS/Fedora/Ubuntu that is the driver you will be most likely using.
  - ex3ndr 8y ago
    For sure it is a very different thing. Local SSD or remote drive? That means a lot for Cassandra.
    - atombender 8y ago
      Kubernetes supports local volumes. With GKE you get local SSDs.

cookiecaper 8y ago

There is no good reason to run non-test database workloads in Kubernetes or Docker. Databases are designed to sit close to the hardware and have a stable, dedicated chunk of resources for a long time, whereas Kubernetes pods are subject to vaporization at any moment. Databases traditionally have fought the operating system to try and maintain enough control to remain performant. Introducing additional layers into this would be dubious at the best of times, but when it's something fundamentally contrary to the application's nature like stateless orchestration, it's pure farce. There could not be an application worse-suited to running in Kubernetes et al than a traditional database. Anyone claiming something that rams this square peg into that round hole is "production ready" is showing that they're an empty husk and shouldn't be trusted near anything important. Note the downvotes already rolling in less than two minutes after I posted this. This subject is a major third rail here. It goes against the agenda of very powerful people and my account has been censured in the past specifically for making this particular argument, that database workloads and Kubernetes don't mix. Keep that in mind when you're asking HN for their experience on this (or any other topic that YC considers critical to the interest of their investments -- they've shown that they're willing to taint the discussion if it gets too dicey).

ownagefool 8y ago
I assume you also consider databases on cloud with mounted block storage as not production ready too?
- cookiecaper 8y ago
  For the record, I have to use my reply allocation sparingly, since usually when I start talking about this I'm mysteriously throttled for long periods. That said -- no, that's not the same thing at all. Barring anomalous conditions, VMs run as long as you keep them running. They won't be reaped and rescheduled onto some other node in the cluster, whether by automated rebalancing processes or by manual `kubectl delete po...` or `kubectl drain`. You can easily set up a VM that will behave more-or-less like conventional hardware if we ignore the perf hit. This is a pretty simple thing. The reason people say you need to make your apps "12 factor" when you go to k8s is because it doesn't work well if your app cares about state. Databases care deeply about state. You can't just kill a DB server and spin up a new one to pick up where it left off. You can't parallelize a DB workload by spinning up 8 little DB nodes. It's not a web server and it just doesn't work like that. Things like CockroachDB exist specifically because normal databases don't work like that. This is where people usually bring up things like annotations, labels, StatefulSets, etc. First, note that the facilities that accommodate stateful workloads are not priorities for Kubernetes and are generally not well-tested or consistent. This wouldn't be a news story or an independent project if they were. Second, please realize you're doing all of that work to try and make Kubernetes do something it's not really designed to do, with potential negative impact on the availability and scheduling processes for the applications that do work well on Kubernetes, when you could just spin a VM and avoid all of these issues entirely. There's no reason to put a production DB on k8s other than cargo culting.
  - DasIch 8y ago
    EC2 instances do go down, EBS volumes fail, hardware fails. Maybe not as frequently as pods in Kubernetes get evicted but at sufficient scale it does occur frequently enough overall that you do need to find ways to handle this automatically without human intervention. Once you’ve achieved that whether your database runs on a VM or in Kubernetes doesn’t make a difference really. Granted, if your not at that scale, running a database in Kubernetes is probably not the best of ideas. That has nothing to do with Kubernetes though, that’s because running a stateful service with decent working backup, recovery and automated failover is difficult in any case. If that’s not your job, you’re probably better off using RDS or something equivalent.

ofrzeta 8y ago

Local volumes don't use the storage driver. Therefore there should be no performance penalty as far as I know. EDIT: There's probably some overhead with the implementation of namespaces and cgroups but I have not found any reliable sources about the quantities. As a side not if you are using memory limits you will have a performance penalty: "Memory and swap accounting incur an overhead of about 1% of the total available memory and a 10% overall performance degradation, even if Docker is not running." (from the Docker docs). This will probably affect databases running in containers as well.

gnufied 8y ago
Yeah this can't be stressed enough. Persistent volumes in Kubernetes are simply bind mounts that exist in your container's namespace. They don't go through docker's storage driver at all. There should be zero penalty for using persistent volumes this way within containers.
- cpuguy83 8y ago
  Sorry, this seems to insinuate that Docker is slowing down (or using slow) storage... this is completely false. Volumes in docker are just host bind mounts. Now depending on your driver/opts (similar to K8s PV backends) this storage can come from anywhere and performance of the volume is totally dependent on the type of storage being used. Now, the container fs that docker sets up is (usually) using a CoW filesystem, and as there is overhead there... but volumes are specifically designed to bypass the container fs.
  - gnufied 8y ago
    Hmm, sorry I did not mean to imply that. It is indeed true that both docker volumes and k8s PVs are just host bind mounts. However I did mean to clarify that - k8s does not uses docker volumes. Whatever performance issues one notices could either be associated with storage provider or container writable layer.

user5994461 8y ago

Running production databases in docker last year: https://thehftguy.com/2016/11/01/docker-in-production-an-history-of-failure/ https://thehftguy.com/2016/11/01/docker-in-production-an-his... Performance issues should be the least of your concern. The docker deamon and container simply hanged because of filesystem issues on CentOS 6. I worked at a company that was dockerizing their stateless services, then planning to dockerize their cassandra databases. Multiple contractors involved. Stateless services failed periodically because of the above issue. Load balancers can failover automatically, broken nodes are rebooted from time to time, limited impact. Noone cared, just a daily deployment routine. I fear the day the cassandra dockerization would happen. They'd lose their entire customer data (hundreds of millions of customers) once two nodes would fail simultaneously, which happened a lot on the stateless services. Thankfully the project never started and the company didn't go bankrupt. Pretty sure employees moved around and plans got canceled. Expect a lot of instability in docker around filesystem, performance issues and race conditions. Low volume stateless web servers don't get to trigger issues much, but databases do.

gnufied 8y ago
I can't possibly hope to change your mind but stability issues with union filesystem driver in docker(part of it was not even docker's problem) and persistent volumes of kubernetes are two very different things. Cassandra running standalone on host(and crashing) is no different from cassandra crashing when running using a PV inside a container. Moreover - all/most Linux distros have switched to using Overlay2 as default driver. If you are running latest version of RHEL/CentOS/Fedora/Ubuntu that is the driver you will be most likely using.
- ex3ndr 8y ago
  For sure it is a very different thing. Local SSD or remote drive? That means a lot for Cassandra.
  - atombender 8y ago
    Kubernetes supports local volumes. With GKE you get local SSDs.