5 ms·

> Running containerized processes makes it easier to run multiple MySQL processes on the same host in different versions and configurations. You're doing it wr

by pmlnr 10y ago

> Running containerized processes makes it easier to run multiple MySQL processes on the same host in different versions and configurations. You're doing it wrong. One doesn't simply run multiple DB servers on the same iron.

saturn_vk 10y ago
Why is that? Each process is using a separate disk on the machine, and if that's the case, where would you get potential problems?
- merb 10y ago
  And what gives you Docker what cgroups would not? I mean they write: Initially, all our clusters were managed by Puppet Of course docker won't actually replace such systems. Uber has the worst enginnering practices i've ever seen. They replaced Puppet with a handcrafted tool that uses docker and call that Dockerizing MySQL, just wow. I wonder when they are at zero money. (and the best thing is probably that they run at AWS and could just use a AMI, but since that isn't written in the article I would not create such assumptions).
  - ec109685 10y ago
    > I would not create such assumptions But you assume they have the worst engineering practices based on a few blog posts?
- falcolas 10y ago
  Memory. Why go to disk for reads in the first place, if your most used data is in memory? You can keep more of the hot data in memory if you use one process instead of two (or five, or ten).
- deleted 10y ago
  [deleted]
vidarh 10y ago
Sure you do. I have installations where the IO bandwidth available from the PCIe based SSDs is 10x what the Postgres databases for a typical customer actually need. There are plenty of circumstances where there's no reason not to run multiple databases on the same physical hardware.
- falcolas 10y ago
  Biggest reason not to run multiple database instances on a single bit of hardware - ram. Say you're running on a VM with 16gb of ram, and one of your clients is particularly active with a 10gb of data and constant usage of that dataset. If you colocate that customer with five other customers using individual processes, at most your star customer will be able to house about 3gb of their dataset in memory. On the flip side, if you have all five customers in one DB instance, then that one customer can house all 10gb of their data in working memory, making it much more performant. You're hamstringing your DB processes by running one instance per set of databases. Run them all in one process, and you will get much more bang for your infrastructure buck.
  - ec109685 10y ago
    The limitation of the approach you are advocating is that all databases would need to replicate from the same master, which isn't always practical.
    - falcolas 10y ago
      Given the choice between performance and ease of replication, the businesses I worked for always chose performance at scale. Perhaps that skews my expectations, but given how much DB performance affects overall app performance, it trumps most other considerations.
  - collyw 10y ago
    Isn't that one advantage of Docker over VM's is that you have access to all of the RAM in a container while you don't in a VM? (I may well be wrong about that).
evanelias 10y ago
I strongly disagree. At large scale, there are huge advantages to running multiple mysqld on one physical host. Facebook does this across their entire DB fleet! Most of their DB hosts run 2 mysqld but some run 8 or more -- it depends on which workload the host is part of. The original motivation was to support multiple hardware generations. PCIe flash cards have become much larger over the years very quickly -- much faster than older cards become end-of-life. The result is that if you're running a large fleet of database hosts for many years, their storage capacity will differ greatly, both between datacenters (e.g. older DCs will have older hardware on average) and eventually within a datacenter that gets a partial refresh. By defining automation configuration that is smart enough to know that some hosts get 1 mysqld and others get 2 mysqld, based on storage / hw generation, there's a much better flash utilization win. This setup also enables faster replacement of failed hosts. Say each host has N mysqld, all part of different pools. If a host fails then you need to hot-copy the data set of each of these, from other replicas in each affected pool, to a new location. The trick is the replacements can copy from N different source hosts, and even go to N different destination hosts as well. This permits massively faster hot copying behavior vs having a single giant mysqld per host. tl;dr it requires a lot of automation but there are very valid reasons for doing this. That said, I would not advocate using Docker to achieve this. It provides little benefit for this scenario. If you're good at calculating mysqld memory usage, you can already just set the buffer pool and per-session buffers to a size that prevents multiple mysqlds from ever swapping. Meanwhile cpu and network rarely are points of saturation for a db host so that tends to work out fine without a quota system. So that leaves i/o as the main resource that the processes will compete for. But Docker cannot provide i/o isolation. From what I understand, Google/YouTube containerizes their data stores, but their systems around containerization are far more advanced than anyone else's. So I assume they've already solved this problem internally, but that doesn't mean the current state of the art in the open source world is up to the task yet.
- derekperkins 10y ago
  YouTube already open sourced their containerized MySQL solution and it works pretty great. http://vitess.io http://vitess.io

by pmlnr 10y ago

saturn_vk 10y ago

Why is that? Each process is using a separate disk on the machine, and if that's the case, where would you get potential problems?

merb 10y ago
And what gives you Docker what cgroups would not? I mean they write: Initially, all our clusters were managed by Puppet Of course docker won't actually replace such systems. Uber has the worst enginnering practices i've ever seen. They replaced Puppet with a handcrafted tool that uses docker and call that Dockerizing MySQL, just wow. I wonder when they are at zero money. (and the best thing is probably that they run at AWS and could just use a AMI, but since that isn't written in the article I would not create such assumptions).
- ec109685 10y ago
  > I would not create such assumptions But you assume they have the worst engineering practices based on a few blog posts?
falcolas 10y ago
Memory. Why go to disk for reads in the first place, if your most used data is in memory? You can keep more of the hot data in memory if you use one process instead of two (or five, or ten).
deleted 10y ago
[deleted]

vidarh 10y ago

Sure you do. I have installations where the IO bandwidth available from the PCIe based SSDs is 10x what the Postgres databases for a typical customer actually need. There are plenty of circumstances where there's no reason not to run multiple databases on the same physical hardware.

falcolas 10y ago
Biggest reason not to run multiple database instances on a single bit of hardware - ram. Say you're running on a VM with 16gb of ram, and one of your clients is particularly active with a 10gb of data and constant usage of that dataset. If you colocate that customer with five other customers using individual processes, at most your star customer will be able to house about 3gb of their dataset in memory. On the flip side, if you have all five customers in one DB instance, then that one customer can house all 10gb of their data in working memory, making it much more performant. You're hamstringing your DB processes by running one instance per set of databases. Run them all in one process, and you will get much more bang for your infrastructure buck.
- ec109685 10y ago
  The limitation of the approach you are advocating is that all databases would need to replicate from the same master, which isn't always practical.
  - falcolas 10y ago
    Given the choice between performance and ease of replication, the businesses I worked for always chose performance at scale. Perhaps that skews my expectations, but given how much DB performance affects overall app performance, it trumps most other considerations.
- collyw 10y ago
  Isn't that one advantage of Docker over VM's is that you have access to all of the RAM in a container while you don't in a VM? (I may well be wrong about that).

evanelias 10y ago

I strongly disagree. At large scale, there are huge advantages to running multiple mysqld on one physical host. Facebook does this across their entire DB fleet! Most of their DB hosts run 2 mysqld but some run 8 or more -- it depends on which workload the host is part of. The original motivation was to support multiple hardware generations. PCIe flash cards have become much larger over the years very quickly -- much faster than older cards become end-of-life. The result is that if you're running a large fleet of database hosts for many years, their storage capacity will differ greatly, both between datacenters (e.g. older DCs will have older hardware on average) and eventually within a datacenter that gets a partial refresh. By defining automation configuration that is smart enough to know that some hosts get 1 mysqld and others get 2 mysqld, based on storage / hw generation, there's a much better flash utilization win. This setup also enables faster replacement of failed hosts. Say each host has N mysqld, all part of different pools. If a host fails then you need to hot-copy the data set of each of these, from other replicas in each affected pool, to a new location. The trick is the replacements can copy from N different source hosts, and even go to N different destination hosts as well. This permits massively faster hot copying behavior vs having a single giant mysqld per host. tl;dr it requires a lot of automation but there are very valid reasons for doing this. That said, I would not advocate using Docker to achieve this. It provides little benefit for this scenario. If you're good at calculating mysqld memory usage, you can already just set the buffer pool and per-session buffers to a size that prevents multiple mysqlds from ever swapping. Meanwhile cpu and network rarely are points of saturation for a db host so that tends to work out fine without a quota system. So that leaves i/o as the main resource that the processes will compete for. But Docker cannot provide i/o isolation. From what I understand, Google/YouTube containerizes their data stores, but their systems around containerization are far more advanced than anyone else's. So I assume they've already solved this problem internally, but that doesn't mean the current state of the art in the open source world is up to the task yet.

derekperkins 10y ago
YouTube already open sourced their containerized MySQL solution and it works pretty great. http://vitess.io http://vitess.io