9 ms·
Scaling Databases at Activision [pdf]
- rejectfinite 4y agoGaming is insane that it even works online in near realtime. Even moreso before. You can buy old wow servers. That was a world.
- alecco 4y ago> ● SQL query compatibility > ● Minimal changes to the application > ● Runs MySQL in backend > ● Kubernetes native > ● Provides Kubernetes operator > We evaluated multiple candidates and chose Vitess https://github.com/vitessio/vitess https://github.com/vitessio/vitess But it's very light on numbers and doesn't show trade-offs or anything.
- Thaxll 4y agoVitess was used for all of Youtube, they moved to Spanner couple of years ago, Vitess is proven and scalable tech.
- tbarn 4y agoKubeCon talks should be out in the next month or so and it has more details.
- edf13 4y ago> Vitess has been a core component of YouTube's database infrastructure since 2011, and has grown to encompass tens of thousands of MySQL nodes. Would love to hear more about this implementation
- XCSme 4y agoMe too, especially, what it does different from if you would manually shard/partition your database. Is it just the automated sharding?
- brycethornton 4y agoThey have a PlanetScale logo on the last slide. Is this just to say "Thanks for your work on Vitess!" or are they using the PlanetScale service for some of this?
- leetrout 4y agoWhen you give up one of the major components of relational databases like foreign keys you pick up lots of performance opportunity. It is amazing what Vitess / PS will help you accomplish but they don't talk a lot about tradeoffs you make to get there (which are similar to what you face with sharding MySQL without additional tooling).
- samlambert 4y agoWe have foreign keys coming soon!
- XCSme 4y agoHi Sam! I stumbled upon your DB service several times, and considered it several times suggesting it as the DB of choice for my customers (uxwizz.com), but the pricing is a bit confusing/hard to estimate in my case, which makes it hard for me to recommend it. Any suggestions?
- candiddevmike 4y agoThings can be really fast when you throw out referential integrity
- berkle4455 4y agoForeign keys are so overrated. Write good code and you don’t have to rely on the DB to enforce referential integrity. It’s the equivalent of running test cases with every db write.
- AdrianB1 4y agoFrom experience you are very right. Also from experience, most teams are not good enough to write the code with that quality that you can give up to referential integrity.
- vecter 4y agoI'm curious, how would you obviate the need for foreign keys with "good" code? Can you provide a toy example or a reference to an article so I can understand better? I've used NoSQL databases a long time ago and currently rely on on good ole PostgreSQL, but I'm having a hard time understanding how "good" code can be a better solution for managing relationships between data than a foreign key.
- jtchang 4y agoI wonder what game this was for. To me this is validation that going the SQL route in almost 99% of new apps is the right way to go. It will be a rare case that you won't be able to scale out given how mature some of these technologies are.
- chrismsimpson 4y agoMongoDB ftw. Even has graph capabilities.
- VectorLock 4y agoIts web scale!
- chillydawg 4y agoDemonware handle the online services for the CoD franchise.
- ultrahax 4y agoThey handle online services for pretty much any Activision-published game, with carve-outs for Blizzard and King - they still do most of their own backend stuff themselves, with some collaborations.
- VirusNewbie 4y agoGithub had DAYS of an outage because they unsuccessfully tried to scale out SQL rather than dealing with NoSql. There are very few things that structured NoSql can't do if you ignore reporting. Once you scale out traditional SQL, you aren't going to be using it for reporting either way though.
- mh- 4y agoThis is a gross mischaracterization of what caused that outage, even just based on the public information from the (excellent) blog post [0] about it. Automating master failover on MySQL without a human in the loop - in the topology GitHub used back then - is risky. 0: https://github.blog/2018-10-30-oct21-post-incident-analysis/ https://github.blog/2018-10-30-oct21-post-incident-analysis/
- rektide 4y ago3 months from start to roll out for tbeir large scale deploy is hella impressive. Just trying to do basic scaling or other simple tasks sounded like an utter nightmare on their old system. Having autonomic computing at their back seems like an obvious win. Tell it your desired state & let the controller do the job. The new paradigm for computer operations is so wonderful.
- angarg12 4y agoInteresting slide deck, but I feel we are missing some of the most interesting details out. Particularly, slides 24 and 25 could be an entire article on their own. * How did you tune Vitess for resiliency? What were the tradeoffs and how was the performance? * How did you migrate from shared in app config to single DB endpoint? (this is an issue we are facing right now) * What do you mean by some queries being too shard aware? How did you optimize queries for efficient routing with Vitess?
- erksch 4y agoIt would have been interesting to know what alternatives they considered. Perhaps this would have been easy to solve using Nomad and TiDB. So it's a story of people who used Kubernetes and MySQL and continued to use Kubernetes and MySQL. The end.
- kokizzu2 4y agoyup lots of companies do agree https://github.com/kokizzu/list-of-tech-migrations https://github.com/kokizzu/list-of-tech-migrations
- erksch 4y agoIt seems that, on that list, it's almost entirely Chinese companies that have migrated to TiDB. Thanks for the list (and the graph), very interesting!
- dweitzman 4y agoAt my previous company when we made the decision to go with Vitess it was because we had confidence that we could accomplish our scalability goals using Vitess for our critical system in a predictable amount of time with a predictable amount of effort and with a safe, incremental, non-disruptive, revertable, testable rollout without downtime. We had no way to have that kind of confidence about migrating off MySQL to an entirely different database (like TiDB). If we'd been addressing a new storage use case from scratch instead of migrating an existing one, the case for using something other than MySQL might have been stronger.
- super256 4y agoI assume Activision actually knows how to scale databases, but has "join queues" for hype? I think everyone knows the infamous Blizzard queues in release week, where you have to wait an hour with only 5000 people in front of you. Happened recently with the OW2 release.
- mh- 4y agoDatabases are not the only contended-for resource in online games. Those queues are usually the result of intentional capacity planning. Overprovisioning is expensive.
- super256 4y ago> Those queues are usually the result of intentional capacity planning. Overprovisioning is expensive. I'd imagine they could spin up new servers on demand.
- Arrath 4y agoCould, but after near 20 years of WoW and however many expansion launches with big ole wait queues each time, they really don't seem to bother.
- ultrahax 4y agoI have to be careful what I say here, given I work for ATVI, I'm former Demonware myself. I do feel safe in saying though that the login queues are _definitely_ not hype. They're designed to constrain a quite-complex distributed system to a login rate that has been load-tested thoroughly, e.g. they know it'll work at that rate.
- cpressland 4y agoAhh, I wanted to go see this talk at KubeCon yesterday but the room was completely full. Glad to see the finer points got posted here.
- LunaSea 4y agoAnyone have more blog posts about video game backend tech like this one?
- ksec 4y agoNone, I do wish the Gaming industry in general produce more contents like these. From Network ( Extreme latency sensitive ) to DB, in game Frame Rate where techniques are not used in Browser and Editors for faster response time. But I think the industry are being pushed hard and dont have much time to produce content for our reading interest.
- geenat 4y agoThese numbers make more sense on hardware from 5 years ago (to be fair, what they were dealing with), but postgres, on modern consumer hardware (Yes, consumer! Ex: KC3000 + Ryzen 7950x), already does over 50k QPS without being overburdened. You could probably squeeze out 100k QPS with faster NVME's. That's like 10 tall servers for their peak QPS? (500k QPS on slide 19) Lots of natural sharding points here too: company, game, usage of data, etc. This is assuming you avoid or go light on expensive features like foreign keys (as Vitess already does). The "scale magic" in Spanner (Big Table) inspired DB's are just hidden automation of traditional sharding- CockroachDB: Sharded indexes, then runs map reduce for you. Scylla/Cassanda: Sharded indexes again, but more limitations for speedups: Eventually consistent. You don't have fast delete- only update (discord uses tombstones). JOIN's are "in app" only. Vitess: Proxy that dismantles/routes your query to the correct server. This scales, but is eventually consistent. JOIN's on co-located data, or "in app".... in simple terms, Vitess is like an externally managed "in app" query parser/router.
- winrid 4y agoMongo's sharding also uses routers like Vitess but can be strongly consistent depending on connection and write settings. Data is divided into chunks and the routers route queries based on where your query lands on chunk ranges.
- aeyes 4y agoI see 500k qps with 30TB data doubling every 2 years on slide 19. In my experience me this isn't close to being possible with a single Postgres box. There is no way around sharding this workload. Vitess has different consistency models, it's not as eventually consistent as you describe.
- achanda358 4y agoWhat kind of queries per second did you see while running on kubernetes, compared to VMs?
- bickett 4y agoThis is an interesting discussion on database performance and scaling. Thanks for sharing!
- ksec 4y agoConsidering [1] mySQL v5.x will EOL this year. And MySQL 8.0 with EOL in 2026. Does anyone knows if MySQL 9.0 will come soon? [1] https://endoflife.software/applications/databases/mysql https://endoflife.software/applications/databases/mysql
- treffer 4y agoThe thing that kills you is IOPs per query. At 30 TBs the hot data set is huge an there is a long tail of queries not loaded into memory. Plus they were saying 10k+ connections per server. So we are probably looking st 500k+ connections. There is also a regular influx on unoptimized / badly written queries as people add features/games etc. Plus you can end up with random spikes. This setup is likely optimized for high read availability. The slides mention the mental overhead of failovers. This is a thing. Try to explain and help each team understand how to do those safely. Downtime are also not an option. So there is quite some information and context (likely) missing. I used to manage a similar scale zoo of MySQL (similar peak QPS, less total data iirc). I am not sure I would subscribe so much on the Vitess as solution side though. Galera and Group Replication can give you hassle free failovers, add some read replicas and you have a really nice DB setup that can scale horizontally amd vertically until you hit the write limitations per server.
- anonymousDan 4y agoI'd be extremely interested to understand what was the bottleneck that made sharding necessary. Storage capacity on a single node? Memory capacity? CPU? Memory bandwidth? I/O bandwidth? Depends on the specific app? Depends on the specific query?
- lastangryman 4y agoVery interesting read. I've worked on game services, and there are a lot of scaling challenges compared to "traditional" apps. Majority of traffic write heavy (many common patterns for scaling read traffic don't help you), massive spikes of launch day players, followed by big drop offs and occasional spikes with content updates. The height of these spikes can also be unpredictable, you're never sure how many players you'll get. The team I worked in had success with DynamoDB which did a lot of the operational heavy-lifting to scale up and down as well handling scale. I would have liked a bit more specifics on their manual sharing solution, and how the vitess implementation differs.
- XCSme 4y ago> PROBLEM Certain Vitess scaling operations fail on FK violation > SOLUTION Temporarily disable FK checks on the target shards I always found disabling FK checks weird, as in, once you disable them, isn't the integrity permanently lost? Or does it check again the integrity once it's reactivated?
- harshitgangal 4y agojust the right time to ask this question. We have an RFC open to add support for foreign keys in Vitess https://github.com/vitessio/vitess/issues/12967 https://github.com/vitessio/vitess/issues/12967