10 ms·
Scala 3 slowed us down?
- spockz 11mo agoFor me the main takeaway of this is that you want to have automated performance tests in place combined with insights into flamegraphs by default. And especially for these kind of major language upgrade changes.
- esafak 11mo agoWhat are folks using for perf testing on JVM these days?
- noelwelsh 11mo agojmh is what I've always used for small benchmarks.
- cogman10 11mo agoFor production systems I use flight recordings (jfrs). To analyze I use java mission control. For OOME problems I use a heap dump and eclipse memory analysis tool. For microbenchmarks, I use JMH. But I tend to try and avoid doing those.
- gavinray 11mo agoasync-profiler
- spockz 11mo agoI use jmh for micro benchmarks on any code we know is sensitive and to highlight performance differences between different implementations. (Usually keep them around but not run on CI as an archive of what we tried.) Then we do benchmarking of the whole Java app in the container running async-profiler into pyroscope. We created a test harness for this that spins up and mocks any dependencies based on api subscription data and contracts and simulates performance. This whole mechanism is generalised and only requires teams that create individual apps to work with contract driven testing for the test harness to function. During and after a benchmark we also verify whether other non functionals still work as required, i.e. whether tracing is still linked to the right requests etc. This works for almost any language that we use.
- deleted 11mo ago[deleted]
- malkia 11mo agoBenchmarking requires a bit of different setup than the rest of the testing, especially if you want down to the ms timings. We have continous benchmarking of one of our tools, it's written in C++, and to get "same" results everytime we launch it on the same machine. This is far from ideal, but otherwise there be either noisy neighbours, pesky host (if it's vm), etc. etc. One idea that we thought was what if we can run the same test on the same machine several times, and check older/newer code (or ideally through switches), and this could work for some codepaths, but not for really continous checkins. Just wondering what folks do. I can assume what, but there is always something hidden, not well known.
- spockz 11mo agoI agree for measuring latency differences you want similar setups. However, by running two versions of the app concurrently on the same machine they both get impacted more or less the same by noisy neighbours. Moreover, by inspecting the flamegraph you can, manually, see these large shifts of time allocation quickly. For automatic comparison you can of course use the raw data. In addition you can look at total cpu seconds used, memory allocation on kernel level, and specifically for the jvm at the GC metrics and allocation rate. If these numbers change significantly then you know you need to have a look. We do run this benchmark comparison in most nightly builds and find regressions this way.
- malkia 11mo agoGood points there - Thanks @spockz!
- esafak 11mo agohttps://en.wikipedia.org/wiki/Hardware_performance_counter https://en.wikipedia.org/wiki/Hardware_performance_counter can help with noisy neighbors. I am still getting into this.
- spockz 11mo agoYes, that can help with detecting how much cpu was actually used during the run. But it doesn’t influence benchmark results. Not sure how exactly to use it for doing subsequent runs and comparing final performance. Then this needs to be extrapolated to final performance in production.
- atbpaca 11mo agoThank you for sharing. Interesting insight on dep libraries.
- game_the0ry 11mo agoI am not a scala fan and do not care for it, but I upvote for the thorough thought process, breakdown, and debugging of the problem. This is how technical blogs should be written. AI aint got shit on this.
- sema4hacker 11mo ago> I was refreshing one of our services. Part of this process was to migrate codebase from Scala 2.13 to Scala 3. My first question was: why?
- pxc 11mo agoScala 3 is sorta a new language, bringing a lot of improvements to the type system: https://docs.scala-lang.org/scala3/new-in-scala3.html https://docs.scala-lang.org/scala3/new-in-scala3.html It also looks like it has some improvements for dealing with `null` from Java code. (When I last used it I rarely had to deal with null (mostly dealt with Nil, None, Nothing, and Unit) but I guess NPEs are still possible and the new system can help catch them.)
- dionian 11mo agoWhy not though the upgrade process from 2.13 to 3 is pretty smooth. And you get all the new language features. I can think of a few that I actually like. I’ll just mention enums because it’s a good example.
- lmm 11mo agoIf you're going to "refresh" a codebase you probably want it to be on the current version of things. Old dependencies rot, like it or not. I don't think there's any timeframe for Scala 2 EOL yet, but new development is happening in 3.
- pjmlp 11mo agoThe only issue I have with Scala 3 is Python envy, they should not have come up with a second syntax, and pushing it as the future. If anything is slowly down Scala 3 is that, including the tooling ecosystem that needs to be updated to deal with it.
- gedy 11mo agoAs a former Scala fan, wow you aren't kidding, wth val month = i match case 1 => "January" case 2 => "February" // more months here ... case 11 => "November" case 12 => "December" case _ => "Invalid month" // the default, catch-all // used for a side effect: i match case 1 | 3 | 5 | 7 | 9 => println("odd") case 2 | 4 | 6 | 8 | 10 => println("even") // a function written with 'match': def isTrueInPerl(a: Matchable): Boolean = a match case false | 0 | "" => false case _ => true
- bdangubic 11mo agomadness :)
- a24j 11mo agoCan you eli5 the madness? And how that relates to python/java?
- deleted 11mo ago[deleted]
- jfim 11mo agoIt's been a while since I touched Scala but wasn't that a thing in previous versions, minus the braces not being present?
- weego 11mo agoYes, that's all just as it was, and in places braces were not required / interchangeable so this is more of an optional compiler choice than a real change
- dmix 11mo ago> After upgrading the library, performance and CPU characteristics on Scala 3 became indistinguishable from Scala 2.13. We had a similar experience moving Ruby 2->3, which has a ton of performance improvements. It was in fact faster in many ways but we had issues with RAM spiking in production where it didn't in the past. It turned out simply upgrading a couple old dependencies (gems) to latest versions fixed most of the issues as people spotted similar issues as OP. It's never good enough just to get it running with old code/dependencies, always lots of small things that can turn into bigger issues. You'll always be upgrading the system, not just the language.
- derriz 11mo agoI was involved in a Scala point version migration (2.x) migration a few years ago. I remember it being painful. Although I recall most of the pain was around having lots of dependencies and waiting for libraries to become available. At the time Scala was on upswing because it had Spark as its killer app. It would have been a good time for the Scala maintainers to switch modes - from using Scala as a testbed for interesting programming-language theories and extensions to providing a usable platform as a general commercially usable programming language. It missed the boat I feel. The window has passed (Spark moved to Python and Kotlin took over as the "modern" JVM language) and Scala is back to being an academic curiosity. But maybe the language curators never saw expanding mainstream usage as a goal.
- hylaride 11mo agoOutside of Android work, has Kotlin really taken over? My understanding is that Java added a lot of functional programming and that took a lot of wind out of Scala's sails (though Scala's poor tooling certainly never helped anything).
- esafak 11mo agoJava's new features are always going to be on paper. The ecosystem, with all its legacy code, is always going to be a decade behind. And if you are starting a new project, why would you pick Java over Kotlin?
- hylaride 11mo agoThat's kind of what I'm asking. I did have a former co-worker write a micro service in Kotlin around 2018. He said that as nice as the language is, the ecosystem was (at the time, not sure how it is today) so utterly dominated by Android development, that he said he wouldn't recommend using it again - half the time he was calling out Java anyways.
- esafak 11mo agoI use kotlin and I do not feel oppressed by Android in any way. And I'd rather call Java libraries from Kotlin than Java. Many have Kotlin wrappers.
- _old_dude_ 11mo agoIn Scala 3, the inline keyword is part of the macro system. When inline is used on a parameter, it instructs the compiler to inline the expression at the call site. If the expression is substantial, this creates considerable work for the JIT compiler. Requesting inlining at the compiler level (as opposed to letting the JIT handle it) is risky unless you can guarantee that a later compiler phase will simplify the inlined code. There's an important behavioral difference between Scala 2 and 3: in 2, @inline was merely a suggestion to the compiler, whereas in 3, the compiler unconditionally applies the inline keyword. Consequently, directly replacing @inline with inline when migrating from 2 to 3 is a mistake.
- dtech 11mo agoKotlin heavily uses the inline keyword basically everywhere, to get rid of lamdba overhead for functions like map. Basically every stdlib and 3rd part library function that takes a lamdba is inlined. In general it's a performance benefit and I never heard of performance problems like this. I wonder if combined with Scala's infamous macro system and libraries like quicklens it can generate huge expressions which create this problem.
- gavinray 11mo agoThe killer is specifically the inlining of macros -- which Kotlin lacks. And not all macros, but just the ones which expand to massive expressions Think template expressions in C++ or proc macros in Rust
- pjmlp 11mo agoThis is one example why being a guest language isn't optimal. They should have made use of JVM bytecodes that allow to optimize lambdas away and make JIT aware of them, via invokedynamic and MethodHandle optimizations. Naturally they cannot rely on them being there, because Kotlin also needs to target ART, JS runtimes, WebAssembly and its own native version.
- dtech 11mo agoKotlin existed before Java 7 and kept support JVM 1.6 for a long time (mainly because of Android) Even then, they benchmarked it, and inlining was still faster* than invokedynamic and friends, so they aren't changing it now JVM 1.8+ is a requirement. * at the expense of expanded bytecode size
- deleted 11mo ago[deleted]
- Kwpolska 11mo agoThe takeaway of upgrading your libraries when upgrading major language and framework versions applies beyond Scala. Especially when the libraries abuse magic language features (and far too many Scala libraries do) or otherwise integrate deep into the framework/language.
- still_grokking 11mo ago> Especially when the libraries abuse magic language features (and far too many Scala libraries do) Would you mind to explain what you mean?
- Kwpolska 11mo agoThe library that caused breakage for the OP is a good example: https://github.com/softwaremill/quicklens https://github.com/softwaremill/quicklens
- still_grokking 11mo agoI still don't get what you want to say. Linking some GitHub repo does not explain anything. What is "abuse", what is "magic" in this context?
- munchler 11mo agoI’m not familiar with Scala’s macro system, but it seems like a big takeaway here is: Be careful with code that invokes the compiler (JIT) at runtime. That seems like it’s asking for trouble.
- dtech 11mo agoMacro's are compile time, there is no runtime codegen. The problem was overly-frequent inlining generating enormous expressions, causing a lot JIT phase and slow execution.
- munchler 11mo agoThank you for the clarification. If I understand correctly, these large expressions are created at compile-time, but the impact isn't felt until JIT occurs in the runtime environment. In that scenario, shouldn't the JIT just run once at startup, though? I'm still not quite understanding how JIT can take so much time in a production environment.
- hunterpayne 11mo agoBecause the jit will let the unoptimized code run a few (hundred) times to take measurements to know what needs to be optimized and how it needs to be optimized. This is a good solution and makes hotspot very effective. The problem is that it happens randomly a few minutes/seconds into the operation of the service. So you randomly have a big pause with the performance hit everytime you run the service. The upside is that this only happens once. But you have to plan for a big performance hit to requests which are unlucky enough to be called at the wrong time.
- pretzellogician 11mo agoAnd this can generally be avoided as well, by doing "warmup" when starting your service (effectively, mock some calls), but before accepting requests.
- phendrenad2 11mo agoControversial opinion: Scala should have gone into maintenance mode a decade ago. They got the language right at the beginning, and a decade of tinkering has just fatigued everyone and destroyed any momentum the language once had.
- instig007 11mo ago> and a decade of tinkering has just fatigued everyone and destroyed any momentum the language once had. it's hard to buy it, considering that many of those "fatigued" moved on Kotlin, led by their managers' bs talking points.
- hunterpayne 11mo agoMany of the Scala projects got people fired. Something the Scala devs largely ignore. Plus Scala support is truly awful even by the low standards of an OpenSource project. Then there is the fact that the Scala specific libraries are largely dead. Scala had/has a lot of promise. But how the language is marketed/managed/maintained really let a lot of people down and caused a lot of saltiness about it. And that is before we talk about the church of type-safety. Scala is a more powerful language than Kotlin. But which do you want? A language with decent support that all your devs can use, or a language with more power but terrible support and only your very best devs can really take advantage of. And I say this as someone writing a compiler in Scala right now. Scala has its uses. But trying to get physicists used to Python to use it isn't one of them. Although that probably says more about the data science folks than Scala. PS The GP is right, they should have focused on support and fixing the problems with the Scala compiler instead of changing the language. The original language spec is the best thing the Scala devs ever made.
- hocuspocus 11mo agoKotlin has become a pretty big and complex language on its own so I'm not sure this is a good counterexample. The fundamental issue is that fixing Scala 2 warts warranted an entirely new compiler, TASTy, revamped macros... There was no way around most of the migration pains that we've witnessed. And at least the standard library got frozen for 6+ years. However I agree that the syntax is a textbook case of trying to fix what ain't broke. Scala 3's syntax improvements should have stuck to the new given/using keywords, quiet if/then/else, and no more overloaded underscore abuse.
- jiehong 11mo ago> After upgrading the library, performance and CPU characteristics on Scala 3 became indistinguishable from Scala 2.13. Checking the bug mentioned, it was fixed in 2022. So, I’m wondering how one would upgrade to scala 3, while keeping old version of libraries? Keeping updated libraries is a good practice (even mandatory if you get audits like PCI-DSS). That part puzzled me more than the rest.
- fn-mote 11mo ago> Checking the bug mentioned, it was fixed in 2022. I was considerably less impressed by the reporting when I finally found out the culprit. Sure it was “Scala 3” … but not really. It was an interaction of factors and I don’t think it would take away from the story to acknowledge that up front.
- mystifyingpoi 11mo agoI'm confused as well, because he wrote > I did it as usual - updating dependencies but later > After upgrading the library, performance and CPU characteristics on Scala 3 became indistinguishable from Scala 2.13. So... he didn't upgrade everything at first? Which IMO makes sense, generally you'd want to upgrade as little as possible with small steps. He just got unlucky.
- gavinray 11mo agoIt would have been a transitive dependency based on the comments about the library being "transparent" and the author unaware it was even used. Pinning specific versions of transitive deps is fairly common in large JVM projects due to either security reasons or ABI compatibility or bugs
- gavinray 11mo agoIf your Maven/Gradle/SBT build specifies a version constraint for a third party lib, updating your Scala or Kotlin version doesn't affect this (For scala-specific libs, there is a bit more nuance, because lib versions contain scala version + lib version, e.g. foolib:2.12_1.0.2 where 2.12 = scala version)
- 11mo ago
- groundzeros2015 11mo agoI know this topic has been beat to death but this is another example of why high level language with super optimizing compiler has had less industry success. If performance is a feature it needs to be written in the code. Otherwise it implicitly regresses when you reorder a symbol and you have no recourse to fix it, other than fiddling to see if it likes another pattern.
- alberth 11mo agoTo be fair, it’s misleading to group Scala (or any JVM language), with other “high-level languages.” The JVM is extremely mature and performant, and JVM-based languages often run 5x (or more) than non-JVM high-level languages like Python or Ruby.
- groundzeros2015 11mo agoThat doesn’t follow. Scala is a high level language and compiler above the JVM. The bug here is a high level one: > Turns out there was indeed a subtle bug making chained evaluations inefficient in Scala 3 I’m comparing with Haskell, Scheme, or even SQl which all promise to compile efficient code from high level descriptions.
- blandflakes 11mo agoThe bug in TFA is hardly a reason that Scala is not a success, though.
- groundzeros2015 11mo agoI didn’t say that. I’m highlighting a specific challenge of getting predictable performance over the lifetime of code. Lower-level languages don’t have this same problem to the same extent. They have other problems Scala doesn’t have.
- still_grokking 11mo ago
- esarbe 11mo agoAwesome language, nice to see others using it. I can thoroughly recommend it. Once of the best languages out there in terms of expressive power.
- xolve 11mo agoThe bug reports linked on softwaremill and scala GitHub's are precise and surprisingly small fixes! It does show Scala's power in expressiveness. Scala is a great language and I really prefer its typesafe and easy way to write powerful programs: https://www.lihaoyi.com/post/comlihaoyiScalaExecutablePseudocodethatsEasyBoringandFast.html https://www.lihaoyi.com/post/comlihaoyiScalaExecutablePseudo... Its a great Python replacement, especially if your project is not tied to ML libraries where Python is defacto, like JS on web.
- hunterpayne 11mo agoThe problem with Scala 3 is that nobody asked for it. The problem with Scala 2 is that the type inference part of the compiler is still broken. Nobody worked on that. Instead they changed the language in ways that don't address complaints. Completely ignore the market and deliver a product nobody wants. That's what happened here. PS Perhaps they should make an actual unit test suite for their compiler. Instead they have a couple of dozen tests and have to guess if their compiler PR will break things.
- thefaux 11mo agoIt's sad but I generally agree. Scala was in my view pretty well positioned for an up and coming language ~2010-15. Not only did the scala 3 rewrite fail to address many of the most common pain points -- compile times and tooling immediately come to mind -- the rewrite took many years and completely stalled the momentum of the project. I have to wonder at this point who is actually starting a new project in scala in 2025. It's really a shame because in many ways I do think it is a better language than anything else that is widely used in industry but it seems the world has moved on.
- zahlman 11mo ago>It's sad but I generally agree. Scala was in my view pretty well positioned for an up and coming language ~2010-15 I used Scala for a bit around that period. My main recollection of it is getting Java compiler errors because Scala constructs were being implemented with deeply nested inner classes and the generated symbol names were too long.
- still_grokking 11mo ago> My main recollection of it is getting Java compiler errors because Scala constructs were being implemented with deeply nested inner classes and the generated symbol names were too long. Sounds like you've used some beta version over 15 years ago. Nothing like described happens in current Scala and it's like that as long as I can think back. Never even heard of such bugs like stated. Coming up with such possibly made up stuff over 15 years later sounds like typical FUD, to be honest.
- scotty79 11mo agoIt's quite impressive that you can swap out major version from under running application and have just one subtle issue.
- still_grokking 11mo agoThe upgrade Scala 2 -> 3 is usually super smooth. The compiler does all the work, you just need to update your build config / dependencies. The only exception is macros: If you used the experimental Scala 2 macros you need to migrate them to the new system which is completely different.
- rr808 11mo agoI'm on Spark Scala 2 project and I hate it. Basically any good Scala dev would never want to work on our ETL projects, so we get second rate Python or Java devs like me who bastardize the language to get anything to work. Most of our new stuff is all pyspark, hopefully we can replace Scala asap.
- still_grokking 11mo agoWhat's so bad about it? Why not try to learn it for good?
- ahZOTERESET 11mo agohttps://www.experimental-history.com/p/bag-of-words-have-mercy-OF.MX.US https://www.experimental-history.com/p/bag-of-words-have-mer...