36 ms·
Lichess gets a big upgrade. It doesn't go as planned
- eppp 4y agoBut what was the JVM tuning? Thats the most interesting part!
- stardenburden 4y agoThe linked blog post has some details: https://lichess.org/@/thibault/blog/lichess-on-scala3-help-needed/2bpotLb0 https://lichess.org/@/thibault/blog/lichess-on-scala3-help-n...
- csense 4y agoAlso see previous HN discussion (top comment is the solution): https://news.ycombinator.com/item?id=33865932 https://news.ycombinator.com/item?id=33865932
- cpleppert 4y agoYeah, literally the first thing you do when running into JVM issues. It was my second initial suggestion in Discord as well.
- xyst 4y agoWould be interesting to see if this application would benefit to switching the JVM from openjdk to Azuls “high performance” JVM. Rather than use a big bang deployment. Setup an A/B test, and gather a sampling of data. I haven’t been able to truly test azul’s claims yet. Java apps that I manage/deploy were on a small scale in regards to requests/sec and I personally didn’t see much difference.
- dikei 4y agoAzul Zing is quit expensive at minimum $15K/year. While Lichess might be able to get it for free under some promotion deals, it would not make sense to tune an OSS projects for such an expensive option since most users won't be able to use it.
- lukhas 4y agoCurrently it's "-Xms30g -Xmx30g -XX:+UseG1GC -XX:+PrintCodeCache -XX:ProfiledCodeHeapSize=500m -XX:NonProfiledCodeHeapSize=500m -XX:NonNMethodCodeHeapSize=24m -XX:ReservedCodeCacheSize=1024m -XX:InitialCodeCacheSize=1024m -XX:ParallelGCThreads=24" Everything after G1GC was suggested by various helpful experts on HN, Discord, by email and other media.
- Buttons840 4y agoIs there any other runtime that needs as much tuning as the JVM? It seems like a failure of the JVM that someone capable of porting thousands of lines of code with little trouble then fails to tune the JVM. It's like, "we wrote all the code, now the hard part starts, tuning the JVM".
- thomashabets2 4y agoJava is a language built up of failed experiments in language design. My full rant about it is https://blog.habets.se/2022/08/Java-a-fractal-of-bad-experiments.html https://blog.habets.se/2022/08/Java-a-fractal-of-bad-experim... It's partially my subjective opinion, but it seems that almost every single language design decision that Java made was, in retrospect, a bad one. Not that I could have done better. It's just that none of it panned out.
- adgjlsfhk1 4y agoThat's not quite fair. They made 1 good design decision (killing pointers).
- thomashabets2 4y agoMy graph of null pointer exceptions per second would disagree with you. It's something so common that "NPE" is a common term used when using Java. But maybe you're saying that the decision was good, but that they failed to actually do it?
- jdlshore 4y ago
- d2049 4y ago> To be able to tell if Scala 3 itself is faster, we would have to rollback to Scala 2 and try it with the proper JVM tuning. I'm not willing to do that, sorry! Once you've tried Scala 3, there's no going back. Pardon my ignorance. Why can't you set up a performance test in both environments and measure the speed?
- typingmonkey 4y agoThat would only confirm that the new lichess version is faster, but not that scala3 is faster then scala2.
- ARandumGuy 4y agoI think it's a matter of effort. Many performance bottlenecks only really show up when you scale up to massive levels. While it's possible to replicate that on a test environment, it takes time and effort to set that up. While doing that is worth it for some applications, I can see why the Lichess developers didn't want to bother.
- hocuspocus 4y agoTime, effort and spare resources that Lichess had better allocate somewhere else. If Thibault enjoys Scala 3 and feels more productive, while the overall performance on the same hardware is approximately equal, it's already a win for the project.
- agilob 4y agoSetting up reliable and reproducible performance environments is as much work as running good quality prod systems. It requires monitoring thing rarely available to monitor, extending prod metrics, creating snapshots, running repeatable tests on different scales. This isn't work for a single person. I'm in a team of 7 working to acquire next customer who will be on average 5x bigger than peaks of our current biggest customer. We've been designing, configuring and creating metrics on EKS since September, alongside training devs and developing performance engineering framework in k6. Lichess doesn't need that much effort and accuracy, but it's still lots of work. Did you know Fargate will schedule you Intel Skylake or Broadwell and there's even 30% difference in performance between them on certain tasks?
- jakub_g 4y agoFor those short on time: "It doesn't go as planned" is explained in the middle of the article: > When everything compiled, I shipped it. And to everyone's surprise, apart from a few bugs I had created while rewriting thousands of lines of code... it worked. It just did. No explosions, no obscure bugs, no memory leak, no performance degradation. That was rather unexpected. ...followed by > Then we saw the JVM CPU usage rise to alarming heights, with unusual patterns. And no obvious culprit in the thread dumps... > it was just the JVM that needed some tuning. Details: https://lichess.org/@/thibault/blog/lichess-on-scala3-help-needed/2bpotLb0 https://lichess.org/@/thibault/blog/lichess-on-scala3-help-n...
- warent 4y ago
- phoronixrly 4y agoAs opposed to what?
- vlovich123 4y agoMachine code. Assembly if you’re weak willed and need some hand holding.
- danielheath 4y agoWritten with a magnet, and a steady hand. Keyboards are a crutch.
- hinkley 4y agoShit the ROMs on the Apollo mission were hand-woven wire looms. 4K wires weaving between and around magnets. Why hold a magnet when you can use a marlin spike?
- deleted 4y ago
- philipwhiuk 4y ago> Significant indentation and optional braces And this is why it's annoying to use Scala. They just decided to rewrite the syntax of the language
- tasuki 4y agoIn Scala, eg 2.13 is a major version. Scala 3 is like a super-major version and still it's backwards compatible with 2.13: the new syntax is fully optional. Which languages don't make any changes to syntax over major versions?
- spiderice 4y agoI'm not against syntax changes, but significant indentation seems like an insane choice to make. That's been a thorn in the side of Python for decades. Even if some people do like the look of it, it doesn't seem worth the tradeoffs.
- civopsec 4y agoIt’s not a problem for Haskell and Scala is closer to Haskell.
- tasuki 4y agoI like significant indentation. If you lose indentation, I want your code to break. It's a feature not a bug! I'm not a big fan of Python, but significant indentation certainly doesn't seem to have hurt its adoption much...
- Myrmornis 4y agoI believe that significant indentation is only a problem for people whose text editor Tab binding doesn't handle it for them? At least, I've used Python for 15 years and can't think of a problem it's caused me -- what problems are you aware of? It's similar to the people who complain about lisp but are pairing up parens manually. I think the solution in both cases is "don't do that" -- there are perfectly good text editors that do these things for you.
- TulliusCicero 4y ago> Improved type inference > We want types, not boilerplate. Sometimes it's best to let the compiler figure out what things are by itself. At first I thought type inference was cool too, but more and more I don't like it. I prefer being able to see a variable's type easily, even when skimming over code. Type inference makes code less readable to me.
- rubyist5eva 4y agoMy IDE can show me inferred types as an overlay in my editor (even in languages with dynamic types like Ruby, jetbrains inference is pretty good when the code is simple), perhaps look for an extension for your preferred editor?
- winter_blue 4y agoIf GitHub could dd support for it, then that would be amazing. As it stands, type inference has a tendency to make code reviews more difficult.
- rubyist5eva 4y agoWhy Github? I'm assuming you mean their code review interface, which is so incredibly lacking I don't even bother with it anymore and I just pull the branch locally and use my IDE.
- pnt12 4y agoIt is lacking, but I haven't found practical ways to review it. I tried the github extension in VS code but the experience was even worse for me. Could you share details about how do you do it? I'm very interested!
- rubyist5eva 4y agoThe IntelliJ editor lets me navigate all changed files, full intellisense and lets me add comments and do approvals directly from my IDE interface.
- tiffanyh 4y ago64 comments a week ago on similar topic (“Lichess on scala3 - help needed”) https://news.ycombinator.com/item?id=33865932 https://news.ycombinator.com/item?id=33865932
- hn_acc_2 4y agoLooks like the top comment was spot on. Were HN users the "Avengers" mentioned in OP?
- chupasaurus 4y agohttps://news.ycombinator.com/item?id=33877670 https://news.ycombinator.com/item?id=33877670 no direct mentions in the updates, though.
- syastrov 4y ago> That's where things got a bit hairy. Lila is built on Play Framework which is not yet ported to Scala 3. > So I forked it and butchered it to remove everything we don't need - which is actually most of the framework. I guess there is hope that Play framework itself will be migrated to Scala 3 and that the dependency on the fork can be removed, but this is taking on a risk - what if there are security updates to the upstream in the mean time?
- tasuki 4y agoI think the plan might be to get rid of the play framework altogether and rather use a couple small, independent libraries to achieve the same. The likelihood of getting security fixes in the future is about the same as getting new security holes created by whatever updates. Butchering away everything they didn't need certainly didn't harm security.
- 12345hn6789 4y agoIsn't light bend more or less falling apart? I thought they announced they would no longer publish changes to play
- hocuspocus 4y agoPlay has been donated to the community one year ago, which has been reasonably active lately. They secured some funding and development efforts have picked up a good pace.
- jbm 4y agoI truly wish I could get Opaque Types (Newtypes?) in Typescript. Great writeup!
- jbm 4y agoI was secretly hoping someone would respond with an insult and the correct way to do Opaque Types in TypeScript, but sadly no such luck. I took a look and found that it is possible to do something similar here (albeit with more heavy lifting) https://www.ferreira.io/posts/opaque-branded-types-in-typescript https://www.ferreira.io/posts/opaque-branded-types-in-typesc...
- shirogane86x 4y agoAs far as I know, the only available approaches are brands and just wrapping it in a one element object (but that's even worse). But the branding is a lot easier if you just use something like zod [1] for all your types. [1] https://github.com/colinhacks/zod#brand https://github.com/colinhacks/zod#brand
- jbm 4y agoThis is great! Thank you for the share.
- stareatgoats 4y agoGot sucked into reading the article because of the "didn't go as planned" but it left my thirst for failure stories unquenched. Should have known. That aside, that thing with opaque types in Scala sounds interesting - but how would the compiler know that the parameter was a UserId and not just a random string?
- lmm 4y agoYou have to explicitly call something like UserId(...), just like with any other type. (You can actually do this technique in C by using a 1-element struct, although it's a lot more cumbersome than Scala since the type system is so much more limited).
- epolanski 4y agoI guess there is a constructor string -> UserID
- Kukumber 4y agoThe power of open source! Congrats to lichess and everyone involved who helped!
- WhiteBlueSkies 4y agoI wonder how node.js would have performed at that scale.
- winrid 4y agoIt wouldn't. They need a shared 20+gb heap. No go for v8.
- exabrial 4y ago-1 for Type inference and `var`. It literally save you no time, increases cognitive load, and is an exposure point for future bugs. Just use Explicit types. They aren't hard, time consuming, or bad. They are your friends.
- jdhdjdbdjdbd 4y agolol no? type inference is great u just suck at programming.
- tareqak 4y agoIs there an automatic refactoring that changes type-inferred variable declarations and changes then into typed declarations? I can see the value when writing code in the early stages of a project to both have flexibility and save keystrokes with type inference to help prototype and explore the problem space. Later on, I can see the benefit of switching to typed variable declarations for readability as time goes on and that part of the code base slips away from my working memory.
- valenterry 4y agoYes, there is. That's one of the great things about Scala's typesystem. It makes those things easy to even write yourself, but there is already something builtin: https://scalacenter.github.io/scalafix/docs/rules/ExplicitResultTypes.html https://scalacenter.github.io/scalafix/docs/rules/ExplicitRe... However, from my experience it's actually better to not annotate most things. I annotate methods are that used very often or are "borders" of my API or interface. Otherwise it is usually not needed because it rarely makes the code more clear and IDEs like IntelliJ actually show you the types inline automatically, so you can always see the return type easily.
- ajnin 4y agoInterestingly the source code base does not seem to have any tests. I guess it speaks of the strength of Scala's type system or the rigorousness of the developer. I tried Scala a long time ago but I didn't like it, it felt every student of Martin Odersky's CS course at the EPFL got to design their own language feature, and they often picked the same one but came up with different implementations.
- shmageggy 4y agoYeah I was kinda surprised how YOLO some of this was. For example > At some point I had to rewrite the Glicko2 rating system from Java to Scala 3 [...] No-one noticed broken ratings, so I suppose it worked. Sheesh. Seems like just the kind of thing for which a couple unit tests would be relatively easy to write and would add a lot of confidence.
- opportune 4y agoAs a few-hours-a-week Lichess user I knew they had to have been yoloing a bit in prod because sometimes they’ll restart servers several times within an hour interval, presumably as they deploy to prod, try to fix them, and redeploy. But I’m honestly shocked it works as well as it does without tests. I mean I’m not a paying customer and I much prefer Lichess to the other chess website so I guess I have little say… Does anybody know if Lichess would accept code contributions to add tests?
- valenterry 4y agoScala certainly doesn't remove the needs for tests. But it reduces it a lot. I would say that it reduces the required number of tests to something between 1% to 10%, depending on the domain. The more "calculations"/"math" the domain contains, the more tests are necessary. But for the "plumbing" which usually makes up for the biggest amount of code, it does wonders. Scala 3 is also quite different compared to the early times. Unfortunately the reputation is still a bit outdated.
- dikei 4y agoThere are some tests, but definitely not a lot for a project of this size https://github.com/lichess-org/lila/search?q=Specification https://github.com/lichess-org/lila/search?q=Specification
- winrid 4y agoThere's a lot of ignorance in this thread around the JVM, which is forgivable I suppose since it's not the hot new thing atm :) The parameters they are tuning are available in any good VM with a JIT. You won't see these tuning knobs with Go because there's no JIT. These are trade offs. Scala is also known to create more garbage than Java. The reason for the CPU spikes are code is getting compiled via JIT which takes CPU. If isn't enough space for JIT compiled code then you'll be recompiling code often, burning CPU. This is similar to database cache tuning. Too little cache and you burn CPU and IOPS. Too much cache and you don't leave enough memory for other operations. These parameters have good defaults but large apps can surpass them. It's important it's not automatic as this could have dire consequences (oom etc).
- kevincox 4y ago> If isn't enough space for JIT compiled code then you'll be recompiling code often, burning CPU. This sounds like a bug to me. If you are out of space for compiled code stop compiling. Or at least start slowly raising the threshold at which you recompile something so that the compilation cost is amortized over time. If your recompiling is using more CPU than was being used before it started something is wrong. You should only be compiling code when it is expected to save CPU long-term.
- winrid 4y agoThat could be. Also this is VM dependent, and there are a handful :) I wonder if it would be easier for them to just use GraalVM.
- aiwv 4y ago> All we need now is for some brave soul to improve Scala 3 support for treesitter, Brave soul is an understatement.