11 ms·
Chess is booming and our servers are struggling
- patrickthebold 4y agoI wonder if they upgraded to Scala 3.
- pkolaczk 4y agoDo they even use Scala? I thought they were using Java (but I may have outdated knowledge). Lichess is written in Scala. Do you have any pointer to an article describing chess.com tech stack?
- cosentiyes 4y agoI haven't seen anything on chess.com's stack before. For gp, previous hn discussion on lichess and scala 3: https://news.ycombinator.com/item?id=33865932 https://news.ycombinator.com/item?id=33865932
- auggierose 4y agoNo, they don't use Scala.
- rjbwork 4y agoIt was a pithy comment on Lichess's recent post on upgrading to Scala 3.
- czx4f4bd 4y agoThe listings on https://www.chess.com/jobs https://www.chess.com/jobs provide some hints. Some mentions of PHP and Java for the backend, Kubernetes, Kafka, Redis, MySQL, Mongo, AWS, and GCP, among other things.
- jacobsenscott 4y agoDid someone say resume driven development?
- Keyframe 4y agoSo, Web-scale tech? </snark> I bet load is more on the everything but the chess engine itself.
- transfire 4y agoWell there’s the bottleneck in a nut shell. Ditch the PHP. At least consider Play if you can’t ditch the Java (I can’t vouch for it, but theoretically it should help.) Otherwise they should consider Elixir/Phoenix as it is easily distributable and/or Rust if they need breakneck speed per thread.
- schiffern 4y agoThe page gave me a server error (appropriate, I suppose). https://archive.is/Sbhnk https://archive.is/Sbhnk
- SanchoPanda 4y agoDoubled since December, and held there? Unreal.
- jsnell 4y agoThe Google Trends graphs support it: https://trends.google.com/trends/explore?date=today%205-y&q=chess https://trends.google.com/trends/explore?date=today%205-y&q=... https://trends.google.com/trends/explore?date=today%205-y&q=chess.com https://trends.google.com/trends/explore?date=today%205-y&q=... None of their explanations sounds at all plausible though.
- wongarsu 4y agoIndividually the reasons given sound weak, but they make some sense if you see it as some self-reinforcing trend: the 2020 chess spike fueled by Corona and Queen's Gambit has put chess back into public consciousness and caused content creators to produce more chess content. That set the scene for the next wave, which was catalyzed by the cheating scandal bringing chess into mainstream media for a couple weeks, and creating narratives for people to follow. This has lead to chess related gifts and chess related social media posts, which reinforces the chess trend. Events like chess boxing probably helped, but are mostly a symptom of chess being more mainstream right now.
- jollyllama 4y agoProbably laid-off devs playing chess all day
- TurkishPoptart 4y agoLichess.org does everything Chess.com does but more, without needing to pay.
- cosentiyes 4y agolichess.org >>> chess.com. I encourage everyone who can afford to donate to do so!
- sebastiansm 4y agoFeature wise I agree with you, lichess is amazing. Chess.com funds a lot of tounaments that make the chess scene a lot more fun and interesting, lichess cant't do that.
- deleted 4y ago[deleted]
- cosentiyes 4y agothey could if everyone donates!
- calderknight 4y agoAnd it respects your freedom
- _Algernon_ 4y agoWhere can I get my soul crushed by Mittens on Lichess?
- pythonlover2153 4y agoPlay a game against Stockfish level 8
- urbandw311er 4y agoI don’t mind paying for something particularly if it means the people who run it can make a living from it —- and hopefully ensure it’s still around in 10 years time.
- teddyh 4y agoEverybody wants to play against Mittens?
- layer8 4y agoTIL: https://esports.gg/news/gaming/what-is-mittens-chess-cat-bot/ https://esports.gg/news/gaming/what-is-mittens-chess-cat-bot...
- jacobsenscott 4y agoI don't really get the whole mittens thing. Chess engines have been better than humans for a long time, and they've been on chess.com forever. I guess the joke of giving it rating of 1 when it is playing at a very high level and a cute picture is good marketing. But getting beaten by an engine isn't particularly fun. It isn't really educational either when it is so much better than you.
- aldonius 4y agoI think it's also that Mittens has a bias towards very aggressive rather than strictly optimal play?
- wongarsu 4y ago> All of this generates data that needs to be written to our databases. Sometimes our systems max out, and just as when someone exercises too hard and has to stop and catch their breath, our servers also become exhausted and need to recover. When that happens, they quit working, and our site and apps become unresponsive Is this a tortured analogy, or are their servers actually crashing under load?
- traverseda 4y agoPresumably they're referring to some kind of periodic database maintenance: https://www.postgresql.org/docs/current/maintenance.html https://www.postgresql.org/docs/current/maintenance.html Alternatively they might be doing something weird involving periodically flushing the database from memory and onto disc. That wouldn't mean "crashing", but likely would mean the database wasn't usable as a database.
- Kranar 4y agoI don't think they're saying it crashes. They're suggesting that they have buffers of data that fill up during periods of high load, and when those buffers get full they have to drop any new incoming requests until the buffers get flushed to the database. During that period of time the site becomes unresponsive.
- adam_arthur 4y agoChess has to be one of the easier games to scale up, no?
- Waterluvian 4y agoGood question. A web chess game is turn based with some basic logic to sync the clock. There are no secrets or complex state like many video games. Seems rather straightforward. The hard part is scaling up everything around it: - Stockfish real-time predictions - anti-cheat monitoring - the moderation tasks any social website needs
- noxvilleza 4y agoAnti-cheat monitoring can happen after the fact though, and Stockfish can run in-browser (there's a JS-compiled version of it).
- Waterluvian 4y agoOh cool. Does it run in the browser for This website? That would scale far better.
- qwertycrackers 4y agoI believe it does. At least, the better version of this site, lichess, runs the analyzer in the browser.
- fbdab103 4y agoA complete chess game should be a very limited amount of data. I am finding conflicting reports that a typical chess game is 40 to 70 moves. That being said, if any service suddenly saw a doubling of users within a month, plenty of sub-optimal architecture decisions can suddenly fall on their face which were otherwise adequate at the previous scale.
- bob1029 4y agoReplace chess with checkers and I think they'd still have ~90% of the same problems. This seems to be more about massive table/data growth than inherent complexity of the core business.
- Waterluvian 4y agoChess found its way into my YouTube feed and that got me back into it. I wonder if this is just some bizarre case of the outcome of some thing like that. “Oops we accidentally put chess in front of everyone for a week and now they’re all playing it.”
- bgirard 4y agoI found that as well. The cheating controversy seemed to have kicked off some interest and then chess started appearing in my YouTube and TikTok feed have started recommending interesting creators. For the first time I've been passively watching content learning chess more formally. One interesting event can kick off feed recommendations and start this fly wheel effect.
- colechristensen 4y agoThe Queens Gambit Netflix show a while back also kicked off a lot of interest.
- hannob 4y agoMe too. I read a couple of articles about the Niemann/Carlssen debate, then I saw a guy talking about chess on youtube, and eventually I created a chess.com account.
- ineedasername 4y agoSame. It also happened for my son (though oddly not my daughter...) and now he's really interested in learning chess and has been self-teaching through internet resources, so I guess that's a good thing.
- slm_HN 4y agoChess.com has the somewhat unique property of being a chess server run by people who aren't good at chess and aren't good at technology. I think they used to hire devs off Fiverr. However they are somewhat good at business as they've convinced thousands of people to pay $17.00 per month for something that can be found for free at Lichess.org. They do pay a ton of money to streamers to use their site exclusively, which is good for chess since it allows more chess players to make a living.
- tester457 4y agoBad at the game, bad at engineering, but good at marketing. There's a lesson in here. Similar to how most of the software that runs the internet is poorly written and has bad fundamentals.
- pictur 4y agothis is a sad fact of life. a great project in terms of engineering can be beaten by a job that is just a wordpress page but great in terms of marketing.
- dan-robertson 4y agoIs the lesson Berkson’s fallacy?
- janalsncm 4y agoThank you for mentioning this, it’s a very interesting thought actually. For the uninitiated, the idea is that a product needs good engineering + good marketing to be successful. Products with only one of them will fail. The idea of Berkson’s fallacy is that even though it seems that good marketing is negatively correlated with engineering (and vice-versa) it’s actually because the baseline level of engineering and marketing is low, and products which happen to excel in one don’t need to excel in the other.
- randmeerkat 4y ago> Bad at the game, bad at engineering, but good at marketing. There's a lesson in here. Similar to how most of the software that runs the internet is poorly written and has bad fundamentals. Maybe now we can get rid of Byzantine tech interview processes and instead just focus on hiring people that are capable of hacking things together.
- lazyvar 4y agoSome previous discussion around scaling their database: https://news.ycombinator.com/item?id=25730778 https://news.ycombinator.com/item?id=25730778
- Benobba 4y agoMove to the cloud. Problem solved.
- enterthematrix 4y agoLichess is both free and much better... so there's always that.
- Kukumber 4y agoThat's hard to believe, they make a shit ton of money Glad i am using and supporting lichess instead
- jacobsenscott 4y agoThey should have no trouble hiring some top notch engineers with experience with high traffic sites right now.
- lukev 4y agoI was going to post that I suspected this was due to some sort of automated or bot activity... a couple months seems like way too short a time for that much organic growth. However, Google Trends also indicates a doubling of interest in chess as a search term over the past three months. Interesting!
- chaosbutters314 4y agolikely due to queens gambit (netflix show) and shotgun king having some influence for both being viral hits
- lofatdairy 4y agoDon't forget the cheating scandal and Carlsen declining to defend the WC title. Both made it into the mainstream news which at the very least caused people to search for chess.
- lukev 4y agoYes, the article covers those (some of them don't fall into the time period under discussion.) There's a few possible drivers that are no doubt helping but it does appear that between one thing and another, chess itself is going viral.
- gronky_ 4y ago> We made a lot of investments in hardware… Sounds bizarre until you remember that over 80% of servers are still on prem
- conor_f 4y agoWow really? Have you got an article or something indicating that? Seems absurd to me that all cloud providers account for less than 20% of servers, either by traffic or sheer volume.
- kaushikc 4y agoLichess is free to use but they do post their expenses and they would appreciate support from the chess playing community.
- alexpotato 4y ago> What does that mean? 250,000+ new accounts are being created each day. People are playing games (16,000 chess moves per second on average) I remember reading how Dropbox and other file sharing sites used hashing to create single instances of popular files and then just used pointers from accounts with that file to save disk space. I would imagine a LOT of chess games probably start out the same and you can therefore store a pointer to where the game is on a subset of potential game starts and then write the difference as you branch out. I may be underestimating the possible start conditions and/or the metadata storage cost (e.g. time in the game of each move doesn't go away).
- codesnik 4y agoyou probably can encode a whole board in a small-ish string, something you can put in a url hash. But chess game is also a history of moves.
- darkr 4y agoPretty sure you could fit it in a 64 bit int/bitmap. Potentially some SIMD optimisations could be done in this direction
- moomoo11 4y agoI have played chess maybe 5 times in my life.. so I have no idea but.. Maybe you could store all the moves in an array of 2 byte ints. I doubt you can have more than 32k moves in a chess game. Assume 10kb per game in memory. You can run 1 million games on 10gb. Store the moves in the end, run your analysis, etc. Or am I missing something?
- playingalong 4y agoI don't play chess, but my finger in the air estimate is that the whole record of the game weighs a few kb at most. I'd venture a guess that all players data and their avatars take more space. Or maybe less, but not much.
- nappy-doo 4y ago
- tzuip 4y agoIt is worth remembering that by playing you give chess.com possibly thousands of games, each of which can be used against you at any time in the future. Just if their cheating algorithm says so and they feel like making it public.
- quadcore 4y agoDont forget tiktok, ive seen a tutorial that changed so much my beginner view of the game it gave me the urge to play some games. It says ultimately what you have to do during a game is, you want to have some positive difference on pawns and then you exchange your pieces then promote then win. How cool is that?
- scrapcode 4y agoI've re-visited the game multiple times but really have trouble engaging with the game. I've tried to read a couple of tutorials here and there on some simple strategies, but it doesn't seem like anything "sticks," and I always feel so clueless after the first few moves as to what I should do. Any suggestions from the HN crowd on what may have provided you an "ah ha!" moment for chess strategy in general?
- oldstrangers 4y agoIf you're not motivated to play the game then there's no reason to continue trying to play it really. If you're not motivated because it's confusing and unrewarding to fail, then persistence and practice is really the only solution.
- spacemadness 4y agoWhat worked for me was the old Chessmaster tutorials done by Josh Waitzkin. It gives you structured practice problems around general principles and strategies. It was kind of addicting the way they structured it and made me want to keep going. What I got out of it were the general fundamentals that allow you to perform an opening without using pure memorization, and how to look for tactical advantages in mid game like forks and skewers. Once I finished it I started easily seeing all the opening mistakes other beginners made and won a bunch of games just following the general principles. I’ve tried some other similar tutorials since but it is still the best I’ve used as an interactive tutorial.
- primitivesuave 4y agoI highly recommend Mato's chess videos on famous games (https://www.youtube.com/@MatoJelic https://www.youtube.com/@MatoJelic), since he keeps them short (3-5 minutes), doesn't talk excessively about every single move, and only shows the interesting variations at critical positions of the game.
- salamo 4y agoSure, I'm not a pro but I've played for a bit so maybe this will help. 1. Choose a chess opening and stick to it. That means you'll always know how to react at the beginning of the game. 2. Counting. If your piece has more defenders than attackers, it's safe. If you're attacking your opponent's piece more times than it's defended, you can control that square. One of the biggest issues I had when starting out was literally just miscounting attackers and defenders. 3. Basic tactics. There are three fundamental tactics in chess: pins, forks, and skewers. Other than that, watching chess Youtubers is pretty helpful. Some good examples are Aman Hambleton's beginner videos[1], and Daniel Naroditsky's "speedruns"[2]. The latter can be slightly more advanced but still very understandable imo. There are other good channels like Hikaru Nakamura's, Gotham Chess, chessbrah, and Eric Rosen but I think the two I suggested are the best. [1] https://www.youtube.com/watch?v=axRvksIZpGc https://www.youtube.com/watch?v=axRvksIZpGc [2] https://www.youtube.com/watch?v=sfjI4jEY58s&list=PLT1F2nOxLHOeyyw85utYJpWtSmxvA-2WR https://www.youtube.com/watch?v=sfjI4jEY58s&list=PLT1F2nOxLH...
- conor_f 4y agoI find it funny that a large business is struggling where an open source product with an incredibly lean team is coping with no issues
- gardenhedge 4y agoI am one of the people now playing chess every day but I don't think it's down to all those reasons listed. In my case, I am using it as a replacement for social media. Previously I'd pick up my phone and go straight to Instagram or some other app. I realised that my time was just wasted scrolling through pictures and videos that has absolutely no value. Chess is something I can open on my app and I can play on the bus or before bed. While it's not the most value adding thing in my life I am enjoying it for now. By the way, I suck at chess. I struggle to beat 900 rated players. Any tips?
- danny_codes 4y agoIf you actually want to get good, you'll need to: (1) play in a focused way, and review your games to see what mistakes you made. Where did you go wrong, and why? (2) study chess theory. Pawn theory, opening theory, etc. This will dramatically improve your play. (3) review games by the pros. Understand why they do what they do. There are a million youtubers who will talk through their choices as they play.
- gardenhedge 4y agoAny specific recommendations?
- kken 4y agoFor some reason, my perception is that random player selection on lichess leads to much stronger opponents than on chess.com in the lower ranks. Probably this is due to the more informed player base on lichess and a large influex of causals on chess.com The problem is that this only fuels the cycle. Casual players are scared away from lichess, possibly extending the rift between both sites...
- threads2 4y agoScrew this company - my poor dad got perma-banned for sandbagging and he has no recourse.
- Floyd001 4y agoContact him for any type of hacking, he is a professional hacker that specializes in exposing cheating spouses, and every other hacking related issues. he is a cyber guru, he helps catch cheating spouses by hacking their communications like call, Facebook, text, emails, Skype, whats-app and many more. I have used this service before and he did a very good job, he gave me every proof I needed to know that my fiancee was cheating. You can contact him on his email to help you catch your cheating spouse, or for any other hacking related problems, like hacking websites, bank statement, grades and many more. he will definitely help you, he has helped a lot of people, contact him on, [Hacker4wise @ gmail com]
- mirabel-stephen 4y ago[flagged]