4 ms·

Is the JVM running out of code/JIT cache space? We had a similar problem recently in vanilla Java: monolith-like servers would seem to eventually "go bad" for

Is the JVM running out of code/JIT cache space? We had a similar problem recently in vanilla Java: monolith-like servers would seem to eventually "go bad" for no real discernable reason after hours/days of uptime. It turned out we needed to increase the amount of code-cache size the JVM was allowed to use. Though supposedly the cache should have kept working (in the LRU-like way many caches operate), we observed that formerly-fine parts of the affected servers also seemed to be unreasonably slow, as if the whole code/JIT caching behavior had been disabled completely.

ulularem 4y ago
To expand a bit more here (and I'll probably slightly flub some terminology): the JVM of course does just-in-time hotspot compilation of frequently-executed code. When it does so, it caches the optimized code for later executions. There are values set for how many times code is executed before it's optimized in this way. A monolith-like server (contains a lot of code to cache), that's been up for hours/days (has eventually triggered many disparate code paths enough to kick-off the optimization) in a new version of a language (speculatively may contain more code to optimize than the previous version) all seem like factors pointing to this potential situation.
- Floegipoky 4y ago
  Do you think that was driven by the code cache repeatedly filling and flushing? And bumping the size resolved it because it wasn't filling anymore? Did you experiment with disabling flushing? My team is scaling up a Java service and we don't have a lot of institutional experience operating such systems, so I'm really interested in JVM tuning "case studies".
- whizzter 4y ago
  This definitely sounds like a plausible explanation if the codegen for Scala3 has changed to enable more dynamism and that in turn makes some functions/patterns far larger. It seems the place to inform them might be on their email or the discord so join up there and suggest it there? Gonna be fun to hear the solution later.
cldellow 4y ago
The jstat output also shows that CCS is at 99.21% usage, which could support this theory. At a previous company, we operated some Scala services and ran into an issue like this. I forget if it was triggered from a JDK update or a Scala update. Scala (at least the 2.x series) generated a lot of classes, so there was a lot of memory pressure on this part of the system. IIRC, we increased the limit and it resolved the issue. I feel like there's a flag you can pass to see JIT invocations, which might also help validate this as the problem. -XX:+PrintCompilation maybe? Caveats: It's been a long time, so my memory may be faulty, and this may not apply any longer.
- xxs 4y ago
  >The jstat output also shows that CCS is at 99.21% usage, which could support this theory. If code cache has run out, the process effectively runs in interpreted mode. I'd wonder however how they would have so much code. Still, they should just run a profiler or any java monitoring tool.
  - munificent 4y ago
    > I'd wonder however how they would have so much code. They probably didn't author that much code, but Scala 3 may have many language features that its compiler desugars to large amounts of generated code. (For example, when C# added support for anonymous functions, they initially did so by compiling each lambda to a generated class with a field for each local variable that the function closed over.)
    - xxs 4y ago
      They still need to use the functions quite a bit to trigger at c1. Normally Java doesn't compile immediately but after enough iterations. Also if scala does that for real, it'd eat the inline budget effectively killing the performance.
      - toast0 4y ago
        > Normally Java doesn't compile immediately but after enough iterations. I've never done perf work with JVM, but ... Is that iteration count reduced over time at all? If not, and from other behavior described, it seems like this could explain the 48 hours of fine, and then spikey garbage. Compilation cache is nearly filled, some bit(s) of code get called often enough to be compiled after 48 hours, finding space in the cache takes lots of cpu, eventually something more useful is evicted, but then you have even more cache pressure because those more useful items will come back in less than 48 hours, and more things will come in as they hit their required 49,50, etc worth of iterations. If it's inexepensive to increase the cache size, seems like something reasonable to try, but the 48 hour period of stability makes testing difficult. I'm assuming the only realistic test system is production, cause that's how it usually is.
michaelt 4y ago
There's a JVM option - -XX:+UnlockDiagnosticVMOptions -XX:+LogCompilation which does what it sounds like. The output might be instructive if you want to monitor the compilation behaviour more closely. And there are tools like JITWatch if you want to get into even more detail.
djexjms 4y ago
In case you didn't know, this was the correct answer. Nice troubleshooting!

ulularem 4y ago

To expand a bit more here (and I'll probably slightly flub some terminology): the JVM of course does just-in-time hotspot compilation of frequently-executed code. When it does so, it caches the optimized code for later executions. There are values set for how many times code is executed before it's optimized in this way. A monolith-like server (contains a lot of code to cache), that's been up for hours/days (has eventually triggered many disparate code paths enough to kick-off the optimization) in a new version of a language (speculatively may contain more code to optimize than the previous version) all seem like factors pointing to this potential situation.

Floegipoky 4y ago
Do you think that was driven by the code cache repeatedly filling and flushing? And bumping the size resolved it because it wasn't filling anymore? Did you experiment with disabling flushing? My team is scaling up a Java service and we don't have a lot of institutional experience operating such systems, so I'm really interested in JVM tuning "case studies".
whizzter 4y ago
This definitely sounds like a plausible explanation if the codegen for Scala3 has changed to enable more dynamism and that in turn makes some functions/patterns far larger. It seems the place to inform them might be on their email or the discord so join up there and suggest it there? Gonna be fun to hear the solution later.

cldellow 4y ago

The jstat output also shows that CCS is at 99.21% usage, which could support this theory. At a previous company, we operated some Scala services and ran into an issue like this. I forget if it was triggered from a JDK update or a Scala update. Scala (at least the 2.x series) generated a lot of classes, so there was a lot of memory pressure on this part of the system. IIRC, we increased the limit and it resolved the issue. I feel like there's a flag you can pass to see JIT invocations, which might also help validate this as the problem. -XX:+PrintCompilation maybe? Caveats: It's been a long time, so my memory may be faulty, and this may not apply any longer.

xxs 4y ago
>The jstat output also shows that CCS is at 99.21% usage, which could support this theory. If code cache has run out, the process effectively runs in interpreted mode. I'd wonder however how they would have so much code. Still, they should just run a profiler or any java monitoring tool.
- munificent 4y ago
  > I'd wonder however how they would have so much code. They probably didn't author that much code, but Scala 3 may have many language features that its compiler desugars to large amounts of generated code. (For example, when C# added support for anonymous functions, they initially did so by compiling each lambda to a generated class with a field for each local variable that the function closed over.)
  - xxs 4y ago
    They still need to use the functions quite a bit to trigger at c1. Normally Java doesn't compile immediately but after enough iterations. Also if scala does that for real, it'd eat the inline budget effectively killing the performance.
    - toast0 4y ago
      > Normally Java doesn't compile immediately but after enough iterations. I've never done perf work with JVM, but ... Is that iteration count reduced over time at all? If not, and from other behavior described, it seems like this could explain the 48 hours of fine, and then spikey garbage. Compilation cache is nearly filled, some bit(s) of code get called often enough to be compiled after 48 hours, finding space in the cache takes lots of cpu, eventually something more useful is evicted, but then you have even more cache pressure because those more useful items will come back in less than 48 hours, and more things will come in as they hit their required 49,50, etc worth of iterations. If it's inexepensive to increase the cache size, seems like something reasonable to try, but the 48 hour period of stability makes testing difficult. I'm assuming the only realistic test system is production, cause that's how it usually is.

michaelt 4y ago

There's a JVM option - -XX:+UnlockDiagnosticVMOptions -XX:+LogCompilation which does what it sounds like. The output might be instructive if you want to monitor the compilation behaviour more closely. And there are tools like JITWatch if you want to get into even more detail.

djexjms 4y ago

In case you didn't know, this was the correct answer. Nice troubleshooting!