9 ms·

Dockerizing MySQL at Uber Engineering

117 points by Walkman 10y ago

Ruphin 10y ago
Maybe this will help dispel some of the myth that Docker is useless for any scenario where you need persistent state. There are still many advantages to wrapping your database in a container, and this post by Uber explains really well how and when to use this technique. We have been running our databases using this same methodology for almost two years now, including automatic initialization for new clusters, auto-configuration of master-slave and federation setup, and automatic failover/recovery. It doesn't solve all the problems, and we still need to manually solve issues (mostly because we use MongoDB and ran into several bugs and 'features' over the years), but for standard operational work it saves a ton of work. Expanding our cluster by adding a new replicated shard is no more than 2 minutes of work in changing the master config, everything else is fully automated.
- memracom 10y ago
  If you would use something like Ansible it would be just as fast because it is fully automated, and there would be no work at all to change a config because Ansible will also customize config files from templates. Chef and Puppet and SaltStack work similarly, just learn how to use your tool of choice.
  - UK-AL 10y ago
    And for a lot of people the tools of choice is docker...
    - toomuchtodo 10y ago
      Picking the wrong tool because its easier never really scales. Docker is not configuration management. Configuration management is not containerization. Disclaimer: DevOps by day. I care because it makes it easier for me (and cheaper for you) to come in and scale things if it was done right in the beginning (instead of me having to revamp your infrastructure to move components to the right tools), but either way I get paid, so it doesn't matter much to me. I'm just trying to help make your life easier.
      - UK-AL 10y ago
        Most configuration management tools are used for managing docker on machines. I don't see them used much for setting up whats inside them. Plus they don't match the convenience of just pulling down an image identical to production for developers(very important). I also don't want to be using puppet on dev machines. Configuration management tools also take longer to run all scripts then just pulling down an image.
        toomuchtodo 10y ago
        > I also don't want to be using puppet on dev machines. Configuration management tools also take longer to run all scripts then just pulling down an image. Which is why you should be provisioning with Vagrant locally to properly replicate production.
- user5994461 10y ago
  > Maybe this will help dispel some of the myth that Docker is useless for any scenario where you need persistent state. Docker is useless for what is not stateless. You can run database in Docker... until the assumptions of Docker will come back to bite you. > There are still many advantages to wrapping your database in a container, and this post by Uber explains really well how and when to use this technique. Well. Run databases in Docker if you've got 3000 clusters and your only concern [that you created yourself] is to run multiple of them on the same hosts.
  - deleted 10y ago
    [deleted]
drej 10y ago
I love these posts for several reasons. 1. People think of companies like Uber primarily as providers of a regular service to customers (similarly to how Airbnb or Netflix is perceived), but it's interesting to see the engineering chops needed to maintain this operation. 2. Given the relative youth of the company, the stack employed is quite modern and often uses cutting edge technologies in production and with real impact on custormers. Everyday users of these technologies usually only get 'textbook' explanations, but blog posts like these allow them to actually find out how these can be used in production, what the caveats are, how they interplay with other parts of the stack etc. 3. Some people complain that when a startup gets bought or implodes, their technology virtually disappears instead of benefitting the wider developer community for learning purposes (I'm not here to argue what the best practice should be, just stating an observation of a common complaint). By continually describing their engineering practices (similarly to Airbnb in their engineering blog) and open sourcing their non-business technology (e.g. go-torch), their experience lives on beyond the life of the company. Last but not least, it build a positive ethos about this company in the developer community.
- zitterbewegung 10y ago
  It seems there is more backlash for Uber's engineering blog posts though and the points seem to make me doubt Ubers engineering chops.
- gaius 10y ago
  but it's interesting to see the engineering chops needed to maintain this operation No. Uber has clearly hired too many engineers and has too little for them to do, leaving them free to come up with Rube Goldberg contraptions like this one. This is not anything any company that isn't swimming in money should even contemplate (and neither is slapping a layer on MySQL and claiming to have invented a "new datastore"). The kids are incharge of the nursery over at Uber and they are in desperate need of a good, adult CTO...
  - toomuchtodo 10y ago
    It doesn't surprise me that I had to scroll so far down in the thread to find this comment. Kudos for saying what needed to be said. EDIT: Make your system as simple as possible, but no more simple than that.
throwbsidbdk 10y ago
It sounds like Uber has a database cluster for roughly every employee?!! Uber has a single product that largely centers around 1 app. How can this be necessary? I have a feeling... That they're dumping realtime GPS data into a bunch of these when they should be using something like Cassandra...
- Pawka 10y ago
  "Single product" which consists of more than X thousands of micro-services.
  - throwbsidbdk 10y ago
    >The entire trip store, which receives millions of trips every day, now runs on Dockerized MySQL databases together with other stores It really does sound like this is part of the issue. Millions of trips is billions of db row per day. A document store is much more amenable to that kind of workload than MySQL
    - simonw 10y ago
      Uber essentually built their own custom document store on top of MySQL. They explain their design and reasons (and why they didn't use Cassandra etc) in this post: https://eng.uber.com/schemaless-part-one/ https://eng.uber.com/schemaless-part-one/
      - throwbsidbdk 10y ago
        Okay thanks, that explains the why but doesn't make it sound less terrible. They essentially built a nosql database on top of MySQL. Forgivable years ago but this was in 2014...
    - randomdata 10y ago
      Which likely explains why they use their MySQL Schemaless engine[1]. 1. https://eng.uber.com/schemaless-part-one/ https://eng.uber.com/schemaless-part-one/
kennethh 10y ago
It does not say so in the article but I guess they have a database cluster for each city or something like that which makes sense since a user do not care about Uber cars in a different city. Do they use GPS to put the Uber car in the right cluster? The users move around more but they are more static so they are centralized somehow?
- throwbsidbdk 10y ago
  It's gotta be the case. Probably choose trip db based on cutting map into squares or hexagons. Dump all location data from drivers and riders into it real time. Sounds like they've got batch jobs running against these databases to discover surge areas and stuff. Probably designed to only hit only 1-2 databases at a time for calculations they do a lot. This really looks like a crap design overall for a company started well in the age of nosql and stream processing.
  - spacemanmatt 10y ago
    I'm going to guess your design assessment is unpopular. You may find that "old-skool" store-and-forward networks are pretty effective for slow-moving dimensions.
- elcct 10y ago
  Disclaimer: I have never used Uber. What if user orders Uber from one city to another? If driver is then in another city, wouldn't be nice to give him some rides in that city or when someone wants to go back to original city? What about cities that are near the border? If user would like to go shopping to another country? (Not uncommon in Europe. Long time ago I used to take local bus from Poland to Germany, do some shopping and go back)
  - icebraining 10y ago
    Sounds fairly simple (which is not to say easy to implement); you keep the car tied to the database of origin for the duration of the ride (other users don't need to know about it while it's occupied), then you can migrate the car to the other DB when it's idle.
pmlnr 10y ago
> Running containerized processes makes it easier to run multiple MySQL processes on the same host in different versions and configurations. You're doing it wrong. One doesn't simply run multiple DB servers on the same iron.
- saturn_vk 10y ago
  Why is that? Each process is using a separate disk on the machine, and if that's the case, where would you get potential problems?
  - merb 10y ago
    And what gives you Docker what cgroups would not? I mean they write: Initially, all our clusters were managed by Puppet Of course docker won't actually replace such systems. Uber has the worst enginnering practices i've ever seen. They replaced Puppet with a handcrafted tool that uses docker and call that Dockerizing MySQL, just wow. I wonder when they are at zero money. (and the best thing is probably that they run at AWS and could just use a AMI, but since that isn't written in the article I would not create such assumptions).
    - ec109685 10y ago
      > I would not create such assumptions But you assume they have the worst engineering practices based on a few blog posts?
  - falcolas 10y ago
    Memory. Why go to disk for reads in the first place, if your most used data is in memory? You can keep more of the hot data in memory if you use one process instead of two (or five, or ten).
  - deleted 10y ago
    [deleted]
- vidarh 10y ago
  Sure you do. I have installations where the IO bandwidth available from the PCIe based SSDs is 10x what the Postgres databases for a typical customer actually need. There are plenty of circumstances where there's no reason not to run multiple databases on the same physical hardware.
neuroid 10y ago
Screenshot from our management console. This looks like an amazing tool.
eggie5 10y ago
A lot of what they are describing in the first half sounds like kubernetes. I wonder if they could do it again would they adopt it...?
- hectaman 10y ago
  agreed. The newer "Pet Set" construct in k8s might simplify this quite a bit: http://kubernetes.io/docs/user-guide/petset/ http://kubernetes.io/docs/user-guide/petset/
  - eggie5 10y ago
    Or as it's been renamed last month to: "StatefulSet" https://github.com/kubernetes/kubernetes/issues/35534 https://github.com/kubernetes/kubernetes/issues/35534 Mostly what I was commenting on was how they are building some type of declarative infrastructure where they define the topology they want and then system will build it. This was screaming k8s to me.
praveenster 10y ago
Is this somewhat similar to the containerizerd MySQL used at YouTube that was open sourced by Google? Vitesse (http://vitess.io/overview/ http://vitess.io/overview/)
falcolas 10y ago
This article shows so little understanding of the software they are using that it frankly makes me a bit mad. Based on my experience as a MySQL DBA, I feel I can safely say that this method of running databases does not scale, and Uber will need to do even more engineering here before too long. Of course, that might be mitigated by the extreme amount of data sharding Uber is doing, but their data will only grow, and this approach will quickly start coming apart at the seams. 1) MySQL requires one file to configure it: my.cnf. This is not exactly a huge amount of configuration which needs to occur. Installs via puppet, Chef or Ansible tend to consist of two commands - one to install the package, and one to write the my.cnf (templates are good so you can use the same command on any sized server). You can add one more command to set up the initial users, should you so desire. 2) Multiple MySQL processes on the same host wastes that host's resources. A single MySQL instance is perfectly capable of running multiple databases, and will respond faster because it will properly allocate the boxes memory according to each DB's usage. Multiple processes will each chomp up the configured bit of memory, not allowing individual databases to use the resources they need. It's always faster to serve data from memory than from disk (even if that disk is an SSD). Worse, under-utilized DB instances will be swapped off to disk, causing even more load and delay as they are swapped back in. 3) Transferring data from one host to another when you need to bring up a stateful process does not scale. Above a few gigs, the transfer process creates significant load on both the source and destination host, and will easily saturate the link between the two. Neither will respond to requests with any alacrity, meaning you typically want to take both hosts out of the active DB pool. 4) The DB restoration process from copying over the raw files can take 10+ minutes, depending on how many dirty pages existed on the source. The restoration process will go faster with logical dumps, but logical dumps will take longer to generate, transfer, and load. To reiterate, this are problems for those companies running at scale, with terabytes of data and dozens (or more) DB servers. When you're running a few GB of data, a DB container is probably going to work fine. Just don't believe for a second that you can scale it the same way you do your web frontend services. It pays to hire experts. How much time and money has Uber sunk into working around their DB, instead of with it?
- liveoneggs 10y ago
  5) configuration and version drift is guaranteed since you have to restart databases to make any changes 5a) ignores MySQL's ability to make many runtime config changes without a restart 6) hard-codes master/slave relationships into the boot process (at least if I'm understanding the config json) 7) adds additional risk (noted as docker crashes) and client issues (noted as "userland proxy" comment) 8) requires additional ops overhead (noted as special care needed for masters) --- But if you already run everything in docker and really want to get rid of puppet, then why not? --- this article links to ansible but not puppet and is the only blog post tagged with either one; I think this really boils down to puppet vs $x
memracom 10y ago
Database servers, whether MySQL or PostgreSQL or NoSQL ones like CouchDB should never be dockerized. This is a use case where it is inappropriate to use Docker at all. This is a case where the db server should use the entire resources of a single server and for managing that server (and its replicas or other cluster members) you use a tool like Ansible or Chef or Puppet. And you need to learn that management tool well, because you will be pushing the limits of server automation with a database server. If you think that a database server is where you put your "application's database" then I question why you have a db server at all. That is the use case SQLITE is for. Database servers are for the use case of storing the business's mission critical data so that all the applications of the business can access needed data. I love Docker and use it a lot, but not for everything.
- aeharding 10y ago
  Eh. For small scale, running a DB in a container and linking the DB storage to a directory on the host's filesystem works well enough. I can't talk about large scale since I don't have first hand experience managing a setup like that.
- marcc 10y ago
  Could you expand on why "database servers... should never be dockerized"? You mention that database servers should have access to all of the resources of the server, but Docker doesn't prevent that. What other reasons would you give someone when they are considering Docker vs Ansible/Chef/Puppet to deploy a database server or cluster?
- UK-AL 10y ago
  There are other benefits to docker than just being able to run multiple processes on the same machine. Putting everything you need to build an image using a dockerfile(Then to version control that) is useful. Developers can also grab docker images for databases for their localmachines without any setup. Also consistency with the rest of the stack.
  - iamatworknow 10y ago
    If you don't need containerization and still want those features you mention why not just use Vagrant?
VLM 10y ago
Most of the assumptions in the comments boil down to its a bad idea if you're resource constrained because you'll get much better performance when everythings on one big server with shared ram. However, possibly, their engineering goal was detecting and isolating hotspots and minimizing debug/downtime effort. And possibly they have an infinite pile of cash or at least they're not resource constrained. In that case it might make sense to fragment massively. For example, lets say database #235 is using 75% of a shared server. Then at least in the very short term migrate the numerous other databases at the container (or image) level to other servers so the overloaded database doesn't flood out every other business system you support. Now the argument is if you ran mysql bare metal then you'd have more memory and maybe only use 25% of that shared server, but sooner or later you'll get a big enough flood of traffic that you'll want to segment off and your operations team understands docker at 2am but not mysql DBA stuff quite as much. Sometimes its nice to have everything use the same standard and everything lives in docker might work. This also has fascinating forensic and QoS implications where you can do interesting snapshot and cloning tricks affecting precisely and exactly one DB at a time.
- exelius 10y ago
  Exactly this. You don't go to containerization because it's cheaper / more efficient, you go to it because it's more flexible and lets you simplify your tech ops into business ops faster. Containers require a LOT of infrastructure / overhead to build -- and don't be seduced by cloud providers that abstract that complexity, if you build a business on containers you WILL have to deal with it. But it does allow you to de-skill a lot of tech ops processes, which helps reliability (so long as your strategies are effective). And you're right; depending on the way they handle their sharding (which may be part of a greater data availability strategy) this might not be so bad. But business goals are driving this, so you have to find a way to make the tech deliver what the business wants.
- korzun 10y ago
  It's almost 2017. The scaling choices are not limited to Docker vs. bare metal. With their budget, you should be able to clone and scale metal servers within minutes if you wanted to. This is like Tumblr spending $50,000 grand a month on their developer AWS instances while not making any money. Cool but not very practical.
amenod 10y ago
> The MySQL data directory is mounted from the host file system, which means that Docker introduces no write overhead If I understand correctly, Docker doesn't introduce any write overhead, mounted from host or not. This is the main difference from VMs - syscalls are just being "forwarded" to kernel. Or am I missing something? Still, I am only arguing with the stated reason. Of course you don't want to save state inside an (ephemeral) container.
- otterley 10y ago
  There's not even a forwarding process. Containers are simply processes running in a set of one or more different "namespaces." A mount point in a container is just an ordinary mount point from the kernel's perspective. In this case, the mount point is relative to the container's root filesystem set via chroot(2).
didip 10y ago
I think the back story here is that they are running MySQL as sharded key-value storage. And at 2300 instances, it makes sense that there would be a few of them dying at any given time. With this setup, using MySQL is pretty much arbitrary, any databases could fill in the role as well. E.g. It got me thinking that Redis+Sentinel could do the job too (if they don't need super durable transactions)
deleted 10y ago
[deleted]
user5994461 10y ago
I guess we should train the new generation better so they don't end up as Uber so... tip for any company who gets incredibly successfull: The first thing one has to do is to drop SQL databases as the main data source. The usual choice is to move to Cassandra. It does have build-in sharding AND backup AND multi-master replication AND multi datacenter support AND performances scale linearly with the number of servers. The [only] other option is ElasticSearch (which has slightly different properties regarding data format and data consistence). You don't need to craft complex custom sharded distributed WTF software to abstract hundreds of cluster of hundreds of databases. Use the right tool for the job, that has that build in.
- pg314 10y ago
  Dropping the RDBM and replacing it with something like Cassandra is fashionable, but sounds like bad advice to me. A quick google gives these stats for Uber[1]: - 8 million users - 160 thousand drivers - 1 million daily rides - 2 billion rides so far The biggest data, as stated in [2], is the trip info. They are storing the trip info as a 20kB JSON blob. If you add a custom data type [3] for a trip to PostgreSQL and encode it efficiently (binary, using deltas), you should be able to do much better than the ~3kB they get by using messagepack and zlib, say 1kB. All that data should easily fit into 4TB. At one million rides per day, one reasonably beefy RDBM server should be able to easily handle that. You can partition the trips table and move the trips that are older than say, 1 month, to a slower disk to save money on disks. That helps with schema migration too: new data uses the new schema, and you use a view on the old data to adapt to the new schema. I very much doubt you need anything much fancier than that. It would help if people really learned their tools (in this case PostgreSQL) instead of jumping to a different technology at the first problem they run into. [1] http://expandedramblings.com/index.php/uber-statistics/ http://expandedramblings.com/index.php/uber-statistics/ [2] https://eng.uber.com/trip-data-squeeze/ https://eng.uber.com/trip-data-squeeze/ [3] https://www.postgresql.org/docs/current/static/xtypes.html https://www.postgresql.org/docs/current/static/xtypes.html
  - user5994461 10y ago
    As a rule of thumbs, putting 5TB of data -that are growing exponentially- in a single box is always a terrible idea. You're gonna hit all kind of limits with the RDBMS software and the special hardware that will be required. Cassandra will handle the sharding automatically, it will have multiple instances with always one available for your applications, it will handle replications across datacenters all around the world, it will have dependable performances that can scale linearly with the specs you give it, it will allow you to do maintenance while online, it will let you add remove and refresh nodes. A bigger RDBMS has none of these qualities. It's a one trick pony that will die when either the hardware or the software will reach a limit, then your whole site will be down. Even if you know what needs to be done to avoid the disaster, you can't do it because the RDBMS is a SPOF and every maintenance you perform is called "downtime". Short term = Postgre/MySQL because it's easier and it gets the job done. Long term = Cassandra because it's dependable.

117 points by Walkman 10y ago

Maybe this will help dispel some of the myth that Docker is useless for any scenario where you need persistent state. There are still many advantages to wrapping your database in a container, and this post by Uber explains really well how and when to use this technique. We have been running our databases using this same methodology for almost two years now, including automatic initialization for new clusters, auto-configuration of master-slave and federation setup, and automatic failover/recovery. It doesn't solve all the problems, and we still need to manually solve issues (mostly because we use MongoDB and ran into several bugs and 'features' over the years), but for standard operational work it saves a ton of work. Expanding our cluster by adding a new replicated shard is no more than 2 minutes of work in changing the master config, everything else is fully automated.

memracom 10y ago
If you would use something like Ansible it would be just as fast because it is fully automated, and there would be no work at all to change a config because Ansible will also customize config files from templates. Chef and Puppet and SaltStack work similarly, just learn how to use your tool of choice.
- UK-AL 10y ago
  And for a lot of people the tools of choice is docker...
  - toomuchtodo 10y ago
    Picking the wrong tool because its easier never really scales. Docker is not configuration management. Configuration management is not containerization. Disclaimer: DevOps by day. I care because it makes it easier for me (and cheaper for you) to come in and scale things if it was done right in the beginning (instead of me having to revamp your infrastructure to move components to the right tools), but either way I get paid, so it doesn't matter much to me. I'm just trying to help make your life easier.
    - UK-AL 10y ago
      Most configuration management tools are used for managing docker on machines. I don't see them used much for setting up whats inside them. Plus they don't match the convenience of just pulling down an image identical to production for developers(very important). I also don't want to be using puppet on dev machines. Configuration management tools also take longer to run all scripts then just pulling down an image.
      - toomuchtodo 10y ago
        > I also don't want to be using puppet on dev machines. Configuration management tools also take longer to run all scripts then just pulling down an image. Which is why you should be provisioning with Vagrant locally to properly replicate production.
user5994461 10y ago
> Maybe this will help dispel some of the myth that Docker is useless for any scenario where you need persistent state. Docker is useless for what is not stateless. You can run database in Docker... until the assumptions of Docker will come back to bite you. > There are still many advantages to wrapping your database in a container, and this post by Uber explains really well how and when to use this technique. Well. Run databases in Docker if you've got 3000 clusters and your only concern [that you created yourself] is to run multiple of them on the same hosts.
- deleted 10y ago
  [deleted]

drej 10y ago

I love these posts for several reasons. 1. People think of companies like Uber primarily as providers of a regular service to customers (similarly to how Airbnb or Netflix is perceived), but it's interesting to see the engineering chops needed to maintain this operation. 2. Given the relative youth of the company, the stack employed is quite modern and often uses cutting edge technologies in production and with real impact on custormers. Everyday users of these technologies usually only get 'textbook' explanations, but blog posts like these allow them to actually find out how these can be used in production, what the caveats are, how they interplay with other parts of the stack etc. 3. Some people complain that when a startup gets bought or implodes, their technology virtually disappears instead of benefitting the wider developer community for learning purposes (I'm not here to argue what the best practice should be, just stating an observation of a common complaint). By continually describing their engineering practices (similarly to Airbnb in their engineering blog) and open sourcing their non-business technology (e.g. go-torch), their experience lives on beyond the life of the company. Last but not least, it build a positive ethos about this company in the developer community.

zitterbewegung 10y ago
It seems there is more backlash for Uber's engineering blog posts though and the points seem to make me doubt Ubers engineering chops.
gaius 10y ago
but it's interesting to see the engineering chops needed to maintain this operation No. Uber has clearly hired too many engineers and has too little for them to do, leaving them free to come up with Rube Goldberg contraptions like this one. This is not anything any company that isn't swimming in money should even contemplate (and neither is slapping a layer on MySQL and claiming to have invented a "new datastore"). The kids are incharge of the nursery over at Uber and they are in desperate need of a good, adult CTO...
- toomuchtodo 10y ago
  It doesn't surprise me that I had to scroll so far down in the thread to find this comment. Kudos for saying what needed to be said. EDIT: Make your system as simple as possible, but no more simple than that.

throwbsidbdk 10y ago

It sounds like Uber has a database cluster for roughly every employee?!! Uber has a single product that largely centers around 1 app. How can this be necessary? I have a feeling... That they're dumping realtime GPS data into a bunch of these when they should be using something like Cassandra...

Pawka 10y ago
"Single product" which consists of more than X thousands of micro-services.
- throwbsidbdk 10y ago
  >The entire trip store, which receives millions of trips every day, now runs on Dockerized MySQL databases together with other stores It really does sound like this is part of the issue. Millions of trips is billions of db row per day. A document store is much more amenable to that kind of workload than MySQL
  - simonw 10y ago
    Uber essentually built their own custom document store on top of MySQL. They explain their design and reasons (and why they didn't use Cassandra etc) in this post: https://eng.uber.com/schemaless-part-one/ https://eng.uber.com/schemaless-part-one/
    - throwbsidbdk 10y ago
      Okay thanks, that explains the why but doesn't make it sound less terrible. They essentially built a nosql database on top of MySQL. Forgivable years ago but this was in 2014...
  - randomdata 10y ago
    Which likely explains why they use their MySQL Schemaless engine[1]. 1. https://eng.uber.com/schemaless-part-one/ https://eng.uber.com/schemaless-part-one/

kennethh 10y ago

It does not say so in the article but I guess they have a database cluster for each city or something like that which makes sense since a user do not care about Uber cars in a different city. Do they use GPS to put the Uber car in the right cluster? The users move around more but they are more static so they are centralized somehow?

throwbsidbdk 10y ago
It's gotta be the case. Probably choose trip db based on cutting map into squares or hexagons. Dump all location data from drivers and riders into it real time. Sounds like they've got batch jobs running against these databases to discover surge areas and stuff. Probably designed to only hit only 1-2 databases at a time for calculations they do a lot. This really looks like a crap design overall for a company started well in the age of nosql and stream processing.
- spacemanmatt 10y ago
  I'm going to guess your design assessment is unpopular. You may find that "old-skool" store-and-forward networks are pretty effective for slow-moving dimensions.
elcct 10y ago
Disclaimer: I have never used Uber. What if user orders Uber from one city to another? If driver is then in another city, wouldn't be nice to give him some rides in that city or when someone wants to go back to original city? What about cities that are near the border? If user would like to go shopping to another country? (Not uncommon in Europe. Long time ago I used to take local bus from Poland to Germany, do some shopping and go back)
- icebraining 10y ago
  Sounds fairly simple (which is not to say easy to implement); you keep the car tied to the database of origin for the duration of the ride (other users don't need to know about it while it's occupied), then you can migrate the car to the other DB when it's idle.

pmlnr 10y ago

> Running containerized processes makes it easier to run multiple MySQL processes on the same host in different versions and configurations. You're doing it wrong. One doesn't simply run multiple DB servers on the same iron.

saturn_vk 10y ago
Why is that? Each process is using a separate disk on the machine, and if that's the case, where would you get potential problems?
- merb 10y ago
  And what gives you Docker what cgroups would not? I mean they write: Initially, all our clusters were managed by Puppet Of course docker won't actually replace such systems. Uber has the worst enginnering practices i've ever seen. They replaced Puppet with a handcrafted tool that uses docker and call that Dockerizing MySQL, just wow. I wonder when they are at zero money. (and the best thing is probably that they run at AWS and could just use a AMI, but since that isn't written in the article I would not create such assumptions).
  - ec109685 10y ago
    > I would not create such assumptions But you assume they have the worst engineering practices based on a few blog posts?
- falcolas 10y ago
  Memory. Why go to disk for reads in the first place, if your most used data is in memory? You can keep more of the hot data in memory if you use one process instead of two (or five, or ten).
- deleted 10y ago
  [deleted]
vidarh 10y ago
Sure you do. I have installations where the IO bandwidth available from the PCIe based SSDs is 10x what the Postgres databases for a typical customer actually need. There are plenty of circumstances where there's no reason not to run multiple databases on the same physical hardware.

neuroid 10y ago

Screenshot from our management console. This looks like an amazing tool.

eggie5 10y ago

A lot of what they are describing in the first half sounds like kubernetes. I wonder if they could do it again would they adopt it...?

hectaman 10y ago
agreed. The newer "Pet Set" construct in k8s might simplify this quite a bit: http://kubernetes.io/docs/user-guide/petset/ http://kubernetes.io/docs/user-guide/petset/
- eggie5 10y ago
  Or as it's been renamed last month to: "StatefulSet" https://github.com/kubernetes/kubernetes/issues/35534 https://github.com/kubernetes/kubernetes/issues/35534 Mostly what I was commenting on was how they are building some type of declarative infrastructure where they define the topology they want and then system will build it. This was screaming k8s to me.

praveenster 10y ago

Is this somewhat similar to the containerizerd MySQL used at YouTube that was open sourced by Google? Vitesse (http://vitess.io/overview/ http://vitess.io/overview/)

falcolas 10y ago

This article shows so little understanding of the software they are using that it frankly makes me a bit mad. Based on my experience as a MySQL DBA, I feel I can safely say that this method of running databases does not scale, and Uber will need to do even more engineering here before too long. Of course, that might be mitigated by the extreme amount of data sharding Uber is doing, but their data will only grow, and this approach will quickly start coming apart at the seams. 1) MySQL requires one file to configure it: my.cnf. This is not exactly a huge amount of configuration which needs to occur. Installs via puppet, Chef or Ansible tend to consist of two commands - one to install the package, and one to write the my.cnf (templates are good so you can use the same command on any sized server). You can add one more command to set up the initial users, should you so desire. 2) Multiple MySQL processes on the same host wastes that host's resources. A single MySQL instance is perfectly capable of running multiple databases, and will respond faster because it will properly allocate the boxes memory according to each DB's usage. Multiple processes will each chomp up the configured bit of memory, not allowing individual databases to use the resources they need. It's always faster to serve data from memory than from disk (even if that disk is an SSD). Worse, under-utilized DB instances will be swapped off to disk, causing even more load and delay as they are swapped back in. 3) Transferring data from one host to another when you need to bring up a stateful process does not scale. Above a few gigs, the transfer process creates significant load on both the source and destination host, and will easily saturate the link between the two. Neither will respond to requests with any alacrity, meaning you typically want to take both hosts out of the active DB pool. 4) The DB restoration process from copying over the raw files can take 10+ minutes, depending on how many dirty pages existed on the source. The restoration process will go faster with logical dumps, but logical dumps will take longer to generate, transfer, and load. To reiterate, this are problems for those companies running at scale, with terabytes of data and dozens (or more) DB servers. When you're running a few GB of data, a DB container is probably going to work fine. Just don't believe for a second that you can scale it the same way you do your web frontend services. It pays to hire experts. How much time and money has Uber sunk into working around their DB, instead of with it?

liveoneggs 10y ago
5) configuration and version drift is guaranteed since you have to restart databases to make any changes 5a) ignores MySQL's ability to make many runtime config changes without a restart 6) hard-codes master/slave relationships into the boot process (at least if I'm understanding the config json) 7) adds additional risk (noted as docker crashes) and client issues (noted as "userland proxy" comment) 8) requires additional ops overhead (noted as special care needed for masters) --- But if you already run everything in docker and really want to get rid of puppet, then why not? --- this article links to ansible but not puppet and is the only blog post tagged with either one; I think this really boils down to puppet vs $x

memracom 10y ago

Database servers, whether MySQL or PostgreSQL or NoSQL ones like CouchDB should never be dockerized. This is a use case where it is inappropriate to use Docker at all. This is a case where the db server should use the entire resources of a single server and for managing that server (and its replicas or other cluster members) you use a tool like Ansible or Chef or Puppet. And you need to learn that management tool well, because you will be pushing the limits of server automation with a database server. If you think that a database server is where you put your "application's database" then I question why you have a db server at all. That is the use case SQLITE is for. Database servers are for the use case of storing the business's mission critical data so that all the applications of the business can access needed data. I love Docker and use it a lot, but not for everything.

aeharding 10y ago
Eh. For small scale, running a DB in a container and linking the DB storage to a directory on the host's filesystem works well enough. I can't talk about large scale since I don't have first hand experience managing a setup like that.
marcc 10y ago
Could you expand on why "database servers... should never be dockerized"? You mention that database servers should have access to all of the resources of the server, but Docker doesn't prevent that. What other reasons would you give someone when they are considering Docker vs Ansible/Chef/Puppet to deploy a database server or cluster?
UK-AL 10y ago
There are other benefits to docker than just being able to run multiple processes on the same machine. Putting everything you need to build an image using a dockerfile(Then to version control that) is useful. Developers can also grab docker images for databases for their localmachines without any setup. Also consistency with the rest of the stack.
- iamatworknow 10y ago
  If you don't need containerization and still want those features you mention why not just use Vagrant?

VLM 10y ago

Most of the assumptions in the comments boil down to its a bad idea if you're resource constrained because you'll get much better performance when everythings on one big server with shared ram. However, possibly, their engineering goal was detecting and isolating hotspots and minimizing debug/downtime effort. And possibly they have an infinite pile of cash or at least they're not resource constrained. In that case it might make sense to fragment massively. For example, lets say database #235 is using 75% of a shared server. Then at least in the very short term migrate the numerous other databases at the container (or image) level to other servers so the overloaded database doesn't flood out every other business system you support. Now the argument is if you ran mysql bare metal then you'd have more memory and maybe only use 25% of that shared server, but sooner or later you'll get a big enough flood of traffic that you'll want to segment off and your operations team understands docker at 2am but not mysql DBA stuff quite as much. Sometimes its nice to have everything use the same standard and everything lives in docker might work. This also has fascinating forensic and QoS implications where you can do interesting snapshot and cloning tricks affecting precisely and exactly one DB at a time.

exelius 10y ago
Exactly this. You don't go to containerization because it's cheaper / more efficient, you go to it because it's more flexible and lets you simplify your tech ops into business ops faster. Containers require a LOT of infrastructure / overhead to build -- and don't be seduced by cloud providers that abstract that complexity, if you build a business on containers you WILL have to deal with it. But it does allow you to de-skill a lot of tech ops processes, which helps reliability (so long as your strategies are effective). And you're right; depending on the way they handle their sharding (which may be part of a greater data availability strategy) this might not be so bad. But business goals are driving this, so you have to find a way to make the tech deliver what the business wants.
korzun 10y ago
It's almost 2017. The scaling choices are not limited to Docker vs. bare metal. With their budget, you should be able to clone and scale metal servers within minutes if you wanted to. This is like Tumblr spending $50,000 grand a month on their developer AWS instances while not making any money. Cool but not very practical.

amenod 10y ago

> The MySQL data directory is mounted from the host file system, which means that Docker introduces no write overhead If I understand correctly, Docker doesn't introduce any write overhead, mounted from host or not. This is the main difference from VMs - syscalls are just being "forwarded" to kernel. Or am I missing something? Still, I am only arguing with the stated reason. Of course you don't want to save state inside an (ephemeral) container.

otterley 10y ago
There's not even a forwarding process. Containers are simply processes running in a set of one or more different "namespaces." A mount point in a container is just an ordinary mount point from the kernel's perspective. In this case, the mount point is relative to the container's root filesystem set via chroot(2).

didip 10y ago

I think the back story here is that they are running MySQL as sharded key-value storage. And at 2300 instances, it makes sense that there would be a few of them dying at any given time. With this setup, using MySQL is pretty much arbitrary, any databases could fill in the role as well. E.g. It got me thinking that Redis+Sentinel could do the job too (if they don't need super durable transactions)

deleted 10y ago

[deleted]

user5994461 10y ago

I guess we should train the new generation better so they don't end up as Uber so... tip for any company who gets incredibly successfull: The first thing one has to do is to drop SQL databases as the main data source. The usual choice is to move to Cassandra. It does have build-in sharding AND backup AND multi-master replication AND multi datacenter support AND performances scale linearly with the number of servers. The [only] other option is ElasticSearch (which has slightly different properties regarding data format and data consistence). You don't need to craft complex custom sharded distributed WTF software to abstract hundreds of cluster of hundreds of databases. Use the right tool for the job, that has that build in.

pg314 10y ago
Dropping the RDBM and replacing it with something like Cassandra is fashionable, but sounds like bad advice to me. A quick google gives these stats for Uber[1]: - 8 million users - 160 thousand drivers - 1 million daily rides - 2 billion rides so far The biggest data, as stated in [2], is the trip info. They are storing the trip info as a 20kB JSON blob. If you add a custom data type [3] for a trip to PostgreSQL and encode it efficiently (binary, using deltas), you should be able to do much better than the ~3kB they get by using messagepack and zlib, say 1kB. All that data should easily fit into 4TB. At one million rides per day, one reasonably beefy RDBM server should be able to easily handle that. You can partition the trips table and move the trips that are older than say, 1 month, to a slower disk to save money on disks. That helps with schema migration too: new data uses the new schema, and you use a view on the old data to adapt to the new schema. I very much doubt you need anything much fancier than that. It would help if people really learned their tools (in this case PostgreSQL) instead of jumping to a different technology at the first problem they run into. [1] http://expandedramblings.com/index.php/uber-statistics/ http://expandedramblings.com/index.php/uber-statistics/ [2] https://eng.uber.com/trip-data-squeeze/ https://eng.uber.com/trip-data-squeeze/ [3] https://www.postgresql.org/docs/current/static/xtypes.html https://www.postgresql.org/docs/current/static/xtypes.html
- user5994461 10y ago
  As a rule of thumbs, putting 5TB of data -that are growing exponentially- in a single box is always a terrible idea. You're gonna hit all kind of limits with the RDBMS software and the special hardware that will be required. Cassandra will handle the sharding automatically, it will have multiple instances with always one available for your applications, it will handle replications across datacenters all around the world, it will have dependable performances that can scale linearly with the specs you give it, it will allow you to do maintenance while online, it will let you add remove and refresh nodes. A bigger RDBMS has none of these qualities. It's a one trick pony that will die when either the hardware or the software will reach a limit, then your whole site will be down. Even if you know what needs to be done to avoid the disaster, you can't do it because the RDBMS is a SPOF and every maintenance you perform is called "downtime". Short term = Postgre/MySQL because it's easier and it gets the job done. Long term = Cassandra because it's dependable.