16 ms·
A bit on scaling chess.com's database
- oli5679 6y agoIf anyone is interested in online chess, https://lichess.org/ https://lichess.org/ is a really interesting project. It is non-for-profit, avoiding freemium upselling and instead relying on user donations Here is a great talk by the founder https://youtu.be/LZgyVadkgmI https://youtu.be/LZgyVadkgmI And here is the open-sourced code. https://github.com/lichess-org https://github.com/lichess-org
- jka 6y agoVery enjoyable talk, thanks for sharing. Funny (amusing) to hear that lichess' competitors would pay professionals not to pay on lichess. Perhaps unsurprising in the era of influencers?
- oli5679 6y agoYes Thibault Duplessis is a very funny guy and has made something great without capturing any personal value!
- x32n23nr 6y ago> Funny (amusing) to hear that lichess' competitors would pay professionals not to pay on lichess That is true, nevertheless, many of the same Grandmasters still play using anonymous accounts. For obvious reasons, we should not point out which accounts these are.
- matsemann 6y agoAs a lichess player myself I have to say I find it a bit tone deaf and annoying that it's peddled in every context chess.com is mentioned.
- newen 6y agoIt’s because lichess is pretty significantly better than chess.com while being free. Creates that extra motivation to make that post.
- jka 6y agoUnder that perspective I'm perhaps guilty of mentioning lichess unnecessarily, but I do think the site is an interesting phenomenon (i.e. providing cutting-edge technology, at no cost to users, without advertising or tracking). Raising awareness of alternatives is good, but yep I think it's also fair to be critical of the way that awareness is generated.
- oli5679 6y agoI think it's an interesting phenomenon and worth spreading some info about, both because of the technical aspects and the philosophy of the creator I prefer. I do the same in person with friends, and if they ignore me, or try it and prefer chess.com then not too much harm done:).
- nmfisher 6y agoI have a chess.com account and wouldn’t have heard about lichess if not for this post.
- chongli 6y agoI think it’s entirely warranted, given that lichess does not have the budget to sponsor big name chess streamers. The only thing that keeps lichess going is the user base.
- dontbesquare 6y agoI was in the Phish vs. phans chess.com match on NYE. I was wondering how this is handled. Fascinating! Thanks for sharing.
- ikonic89 6y agoGlad you liked it :)
- awbraunstein 6y agoI was also there, and when it wasn't working at first I assumed the DB was totally overloaded. Glad you got it working in the end!
- drw 6y agoThe team had to selectively throttle access to that one game and change things around to get through the overload. I'm trying to pun "paged" with "Phish" and it doesn't work. But ops was definitely on deck, on the holiday.
- hashkb 6y agoIt's worse than that. They bailed on voting and had a human watching the chat for audience moves.
- zappo2938 6y agoThey mention the huge bump in traffic from Netflix's "The Queen's Gambit". However, it is worth seeing the Google Trend for searches for both the show and chess since they wrote the blog post. Initially there was correlation between interest in the show and the game, however, as interest in the show wained, interest in the game remains steadily high.
- ikonic89 6y agoYou are 100% correct. We expected to see a drop in activity after the interest in show reduced, but we're still hitting records day after day. Crazy.
- dilyevsky 6y agoMy theory is online streaming is more of a factor than the show
- ajkjk 6y agoyep. From what I've seen: it's because a bunch of chess Twitch and Youtube people have been building up communities + meme-scapes around the game. QG added a bunch of new people who found the online community and got into it -- especially because with the pandemic people have been wandering the internet searching (unintentionally) for new hobbies to get invested in.
- conradfr 6y agoHaving not played in years and like everyone else I guess I went to chess.com after watching the Queen's Gambit, which they mention was an even bigger peak than lockdown, and I was impressed by the site: you can play immediately without registering and the user interface is nice and intuitive. It's very refreshing.
- Buttons840 6y agohttps://lichess.org/ https://lichess.org/ is also very good. You can play without registering, all features are available to everyone for free, and the site is open source. It's a good clean website without ads or tracking. (I used to be bitter about chess.com, thinking they're just cashing in on their domain name and charging for features because they can. And that may be, I don't know. But I have seen them organize some good chess events, so they might not be all bad. I think they pay streamers to use their site though, a practice I'm not too happy about.)
- conradfr 6y agoI think you mean .org and yeah that seems alright and the open-sourcing is interesting (resisting trolling on node ;). It's all about SEO, chess.com is the first result on Google and DDG for "chess".
- Buttons840 6y agoThanks for the correction: https://lichess.org/ https://lichess.org/
- ptudan 6y agoWhat's so bad about influencer marketing? It is the norm these days. Inevitably competition will do it if you don't.
- oli5679 6y agoAs an online chess player I find it a bit shady. Lichess.org has a better UX and featureset for free than a premium chess.com subscription, but many of my friends watch steamers --> sign up to chess.com and get a worse expereince.
- forinti 6y ago> No foreign keys, many things were done in the code itself (filtering, sorting, etc, to make sure the DB only ever uses the most efficient indexes) This normally is terrible advice. You have to be sure that you really need it. Too often I see developers do this needlessly.
- ikonic89 6y agoYeah, I can agree with that. You really need to know what you're doing. We have benchmarks for most of such "micro-optimizations", though now we can know how things are going to behave before we even make them, just from the experience. Definitely not something I'd suggest on an average traffic website.
- yardstick 6y agoikonic89: how large is the database? The charts showed disk utilisation % but would be interested to know raw sizes.
- ikonic89 6y agoSorry for the late reply :) Looks like we have between 20T and 25T of data, combined across all MySQL databases. And somewhere in the region between 60B and 80B records. Edit: this is just for data that's used all the time. Almost nothing is dead, just sitting around.
- david_shaw 6y agoIt's absolutely astounding that you can go on Twitch and see Grandmasters playing speed chess at pretty much any time of day. I agree with other posters who've said that while Queen's Gambit may have sparked interest, it's streamers -- professional chess players or not -- who are keeping interest high. I'm a big fan of chess, and it's awesome to see so many of my friends suddenly become interested in the game!
- hashkb 6y agoAnd it's Phish's "Dinner and a Rematch" that annihilated their infrastructure on 12/31/20 and is no doubt bringing about these projects.
- np_tedious 6y agoWould've been nice to see something about this in the article
- hashkb 6y agoIt was a catastrophe. Not surprised they gloss over it. Plus Phish is more polarizing than Trump so it's dangerous to associate with your brand. Songs are longer than chess matches.
- TedShiller 6y agoThe first mistake is using MySQL.
- corobo 6y agoSaying that in response to a 12 year old database they've built an empire off makes you look silly, not them.
- TedShiller 6y agonot really
- corobo 6y agoFair. I'm not sure why I replied honestly, and so hostility.. It would have cost me nothing to just scroll on by. Sorry mate.
- gregwebs 6y agoThis was a great solution and execution under pressure. You might consider TiDB in the future when you have some breathing room to consider a new operational model. It is MySQL compatible, open source, and can scale writes horizontally as traffic increases by just adding additional nodes. Disclaimer: I work at PingCAP, the creator of TiDB.
- moehm 6y ago> One of things that has really helped our DB scale is the fact that we’ve moved a lot of sorting, merging, filtering from the DB into the code itself. This goes against everything I have learned until now. Maybe an experienced DBA can explain when moving stuff out of the database is helpful? I mean sorting, merging and filtering is the core discipline of a RDBMS and many, many man hours went into it. How and when can a custom version be better?
- bob33212 6y agoIt is easier to scale up application servers than DBs. You want the DB to spend its CPU cycles on what its real value which is ACID.
- dbmikus 6y agoIf you have 1m clients making DB requests and are hitting performance constraints you could do any of: 1. Add more ram, CPU, etc to the DB host 2. Create DB read replicas for higher read volume 3. Shard the DB (reads and writes) 4. Offload logic to stateless clients, who you can easily scale horizontally So option 4 is reasonable if you don't want to or can't do the other options.
- pupdogg 6y agoI would need to know a little more about those 1M requests before making any of your points valid. If 1M are solely reads, you can do real-time transient offload onto Redis without calling it cache and breaking your bank.
- dbmikus 6y agoTrue. I would rephrase option 2 as "cache and/or replicate your query results somewhere".
- BurningFrog 6y agoA single DB machine can easily become a performance bottleneck. It doesn't matter if it sorts 1000x faster than client code, when there aren't cycles left to perform this (in theory) very performant sorting. This might not be your actual problem, but I've seen it hit a completely oblivious dev team before :)
- electricshampo1 6y agoI think it is unfortunate that single node/non scalable databases are still being used, which then leads people to have to do gymnastics to mitigate their limitations (no write scaling, etc). If they used something like Spanner, they would probably not have needed to do any of this and could instead focus on their core business.
- tyingq 6y agoI'd be interested in what the estimated monthly bill for them would be using Cloud Spanner.
- rco8786 6y ago> it was in a fairly decent state for a database that started its life over 12 years ago. Not many unused/redundant indexes, the ones that were there were mostly good. Ah, to work on a service whose requirements have not changed in 1500 years :P. J/k (kinda), enjoyed the read thanks!
- 55555 6y agoTheir business has grown a lot due to The Queen's Gambit. It's pretty cool when an outsized force grows your business like this!
- mst 6y agoAs a minor note, the auto-increment thing can often be handled by having one of the machines only generate even primary keys and the other only generate odd. Do keep an eye out for id exhaustion if you try that though, since you just halved your headroom when running 100% on one or the other.
- tutfbhuf 6y agoThere are "new generation" databases, some of them MySQL 5.6 compatible (e.g. Vitess) with the ability to massively scale in terms of read/write QPS (queries per second). In database related podcasts you will hear that QPS is considered to be a solved problem nowadays.
- ikonic89 6y agoWe did test Vitess couple of years ago, just decided not to go with it. There were a lot of reasons for that, maybe a topic for another post.
- tutfbhuf 6y agoVitess is just one example, there are others. And it depends, some teams might be able to do some work to get along with MySQL, like chess.com. As in most cases, there is no silver bullet here.
- carlsborg 6y agoNice engineering blog post. Thanks for sharing.