4 ms·
This article shows so little understanding of the software they are using that it frankly makes me a bit mad. Based on my experience as a MySQL DBA, I feel I ca
by falcolas 10y ago
This article shows so little understanding of the software they are using that it frankly makes me a bit mad. Based on my experience as a MySQL DBA, I feel I can safely say that this method of running databases does not scale, and Uber will need to do even more engineering here before too long. Of course, that might be mitigated by the extreme amount of data sharding Uber is doing, but their data will only grow, and this approach will quickly start coming apart at the seams.
1) MySQL requires one file to configure it: my.cnf. This is not exactly a huge amount of configuration which needs to occur. Installs via puppet, Chef or Ansible tend to consist of two commands - one to install the package, and one to write the my.cnf (templates are good so you can use the same command on any sized server). You can add one more command to set up the initial users, should you so desire.
2) Multiple MySQL processes on the same host wastes that host's resources. A single MySQL instance is perfectly capable of running multiple databases, and will respond faster because it will properly allocate the boxes memory according to each DB's usage. Multiple processes will each chomp up the configured bit of memory, not allowing individual databases to use the resources they need. It's always faster to serve data from memory than from disk (even if that disk is an SSD). Worse, under-utilized DB instances will be swapped off to disk, causing even more load and delay as they are swapped back in.
3) Transferring data from one host to another when you need to bring up a stateful process does not scale. Above a few gigs, the transfer process creates significant load on both the source and destination host, and will easily saturate the link between the two. Neither will respond to requests with any alacrity, meaning you typically want to take both hosts out of the active DB pool.
4) The DB restoration process from copying over the raw files can take 10+ minutes, depending on how many dirty pages existed on the source. The restoration process will go faster with logical dumps, but logical dumps will take longer to generate, transfer, and load.
To reiterate, this are problems for those companies running at scale, with terabytes of data and dozens (or more) DB servers. When you're running a few GB of data, a DB container is probably going to work fine. Just don't believe for a second that you can scale it the same way you do your web frontend services.
It pays to hire experts. How much time and money has Uber sunk into working around their DB, instead of with it?
- liveoneggs 10y ago5) configuration and version drift is guaranteed since you have to restart databases to make any changes 5a) ignores MySQL's ability to make many runtime config changes without a restart 6) hard-codes master/slave relationships into the boot process (at least if I'm understanding the config json) 7) adds additional risk (noted as docker crashes) and client issues (noted as "userland proxy" comment) 8) requires additional ops overhead (noted as special care needed for masters) --- But if you already run everything in docker and really want to get rid of puppet, then why not? --- this article links to ansible but not puppet and is the only blog post tagged with either one; I think this really boils down to puppet vs $x
- falcolas 10y ago> But if you already run everything in docker and really want to get rid of puppet, then why not? And you know, I wouldn't argue too much at the thought of running MySQL out of a container. It does avoid a lot of basic issues around upgrades, versioning, stack split, etc. It's the act of treating it like all other containerized software which gets my panties in a twist. Stateful systems can't be treated like stateless systems if you want to maintain anything resembling reasonable uptime and performance.
- merb 10y ago> But if you already run everything in docker and really want to get rid of puppet, then why not? Docker does not replace puppet and they clearly wrote that they developed a System just for that.
- agentgt 10y agoI find it ironic that one of Uber's original impetus for migrating off Postgres (which I still have serious doubts about) was not being able to "trust" the system [1]. Given my limited failed experience with playing with docker + postgres (and various volume stores including flocker) I have continued doubts about Uber's choices. I suppose they are having success but I wonder at what cost. Specifically the quote on [1]: Finally, our decision ultimately came down to operational trust in the system we’d use, as it contains mission-critical trip data. Alternative solutions may be able to run reliably in theory, but whether we would have the operational knowledge to immediately execute their fullest capabilities factored in greatly to our decision to develop our own solution to our Uber use case. This is not only dependent on the technology we use, but also the experience with it that we had on the team. [1]: https://eng.uber.com/schemaless-part-one/ https://eng.uber.com/schemaless-part-one/