3 ms·

>The jstat output also shows that CCS is at 99.21% usage, which could support this theory. If code cache has run out, the process effectively runs in interpret

by xxs 4y ago

>The jstat output also shows that CCS is at 99.21% usage, which could support this theory. If code cache has run out, the process effectively runs in interpreted mode. I'd wonder however how they would have so much code. Still, they should just run a profiler or any java monitoring tool.

munificent 4y ago
> I'd wonder however how they would have so much code. They probably didn't author that much code, but Scala 3 may have many language features that its compiler desugars to large amounts of generated code. (For example, when C# added support for anonymous functions, they initially did so by compiling each lambda to a generated class with a field for each local variable that the function closed over.)
- xxs 4y ago
  They still need to use the functions quite a bit to trigger at c1. Normally Java doesn't compile immediately but after enough iterations. Also if scala does that for real, it'd eat the inline budget effectively killing the performance.
  - toast0 4y ago
    > Normally Java doesn't compile immediately but after enough iterations. I've never done perf work with JVM, but ... Is that iteration count reduced over time at all? If not, and from other behavior described, it seems like this could explain the 48 hours of fine, and then spikey garbage. Compilation cache is nearly filled, some bit(s) of code get called often enough to be compiled after 48 hours, finding space in the cache takes lots of cpu, eventually something more useful is evicted, but then you have even more cache pressure because those more useful items will come back in less than 48 hours, and more things will come in as they hit their required 49,50, etc worth of iterations. If it's inexepensive to increase the cache size, seems like something reasonable to try, but the 48 hour period of stability makes testing difficult. I'm assuming the only realistic test system is production, cause that's how it usually is.
    - xxs 4y ago
      >If it's inexepensive to increase the cache size, seems like something reasonable to try absolutely, you just never let that thing drop below 80%. My advice would be to also run C2 compilation at 100-1000 cycles (e.g. -XX:CompileThreshold=100) - it's a much slower startup (but they do have lots of CPUs), and it may not generate a good perf guided code but they will have a good idea how much code cache to dedicate.
xendo 4y ago
They are using JDK17, which segments the cache into profiled and non-profiled code. Both at around 120MB by default, it’s not that difficult to hit.

by xxs 4y ago

munificent 4y ago

> I'd wonder however how they would have so much code. They probably didn't author that much code, but Scala 3 may have many language features that its compiler desugars to large amounts of generated code. (For example, when C# added support for anonymous functions, they initially did so by compiling each lambda to a generated class with a field for each local variable that the function closed over.)

xxs 4y ago
They still need to use the functions quite a bit to trigger at c1. Normally Java doesn't compile immediately but after enough iterations. Also if scala does that for real, it'd eat the inline budget effectively killing the performance.
- toast0 4y ago
  > Normally Java doesn't compile immediately but after enough iterations. I've never done perf work with JVM, but ... Is that iteration count reduced over time at all? If not, and from other behavior described, it seems like this could explain the 48 hours of fine, and then spikey garbage. Compilation cache is nearly filled, some bit(s) of code get called often enough to be compiled after 48 hours, finding space in the cache takes lots of cpu, eventually something more useful is evicted, but then you have even more cache pressure because those more useful items will come back in less than 48 hours, and more things will come in as they hit their required 49,50, etc worth of iterations. If it's inexepensive to increase the cache size, seems like something reasonable to try, but the 48 hour period of stability makes testing difficult. I'm assuming the only realistic test system is production, cause that's how it usually is.
  - xxs 4y ago
    >If it's inexepensive to increase the cache size, seems like something reasonable to try absolutely, you just never let that thing drop below 80%. My advice would be to also run C2 compilation at 100-1000 cycles (e.g. -XX:CompileThreshold=100) - it's a much slower startup (but they do have lots of CPUs), and it may not generate a good perf guided code but they will have a good idea how much code cache to dedicate.

xendo 4y ago

They are using JDK17, which segments the cache into profiled and non-profiled code. Both at around 120MB by default, it’s not that difficult to hit.