7 ms·

More evidence for problems in VM warmup

87 points by ltratt 4y ago

nightpool 4y ago
I appreciate that these issues definitely complicate research and benchmarking of VM performance, but I'm confused about this question: "If you're a normal user, the results suggest that you're often not getting the performance you expect". Is that true? Most VMs are not simple, predictable systems that would ever reach a "steady state"—they're dynamic systems that are constantly executing different parts of the codebase and taking wildly different codepaths at different times. V8 is a great example of this—the type of code executed on a page changes wildly depending on what you're doing and what actions you take. Why would we even want to optimize it for "reaching a steady state", when different parts of the codebase may be more or less useful at different times? It seems more important to me to work on optimizations that can allow us to deoptimize parts of the codebase that we don't think we will be useful again, to save on memory, even if it involves sacrificing this theoretical notion of a "steady state"
- chrisseaton 4y ago
  > Most VMs are not simple, predictable systems that would ever reach a "steady state"—they're dynamic systems that are constantly executing different parts of the codebase and taking wildly different codepaths at different times. V8 is a great example of this—the type of code executed on a page changes wildly depending on what you're doing and what actions you take. That is already the exact point being made by Laurie and Edd's research. > Why would we even want to optimize it for "reaching a steady state", when different parts of the codebase may be more or less useful at different times? A steady state for whatever the application is doing right now. Not a single, statically known steady state, without considering what it's doing. > It seems more important to me to work on optimizations that can allow us to deoptimize parts of the codebase that we don't think we will be useful again, to save on memory, even if it involves sacrificing this theoretical notion of a "steady state" I think that's a completely solved problem isn't it? You can GC a compiled method. What more do you want?
  - dan-robertson 4y ago
    It smells like a caching problem rather than a gc problem. Surely the compiled method would be kept live by the pointer from its interpreted cousin or from its class’s vtable or whatever.
    - chrisseaton 4y ago
      It’s a kind of weak pointer.
      - dan-robertson 4y ago
        Yeah I see the comparison, it just feels… weak. With weak pointers there isn’t really much of a ‘gc keeps around objects that are used a lot’ thing going on whereas that fits much more nicely for a cache and for this situation.
- hedora 4y ago
  After thinking about these issues for over a decade, and having the rug repeatedly pulled out from under me by Java, I've come to the conclusion that the easiest path forward is to stick code snippets into godbolt, or to disassemble optimized binaries from ahead of time compiled languages. (I didn't say doing this is easy; just that it is easier than reasoning about a JIT)
  - PathOfEclipse 4y ago
    What's the difference in practice between a JIT and an AOT compiler that supports optimizations based on runtime analysis (I forgot the technical term for that. Profile-guided optimization, maybe?)? I think the real issue is, the higher level the language, the harder it is to reason about what your code will actually get compiled down to. If you want predictable performance in Java, or any high-level language, in my experience you have to essentially write C code in that language, or at least get as close as possible. In Java, that usually means: * sticking to primitives or arrays of primitives as much as possible. The language has no support for user-defined value types! * Sticking to regular for loops, while loops, and other similar control structures. * Using offheap memory via Unsafe or similar API when feasible.
    - jeffbee 4y ago
      The things that most Java applications need to do to reduce the variance of runtime outcomes are sadly a little more basic: use a modern VM, set the compilation flags appropriately, and make sure the code caches are large enough to hold all your hot functions.
      - PathOfEclipse 4y ago
        My experience leads me to the exact opposite conclusions. For example, your code cache is either big enough or it's not. It's a pretty binary situation and a simple checkbox to ensure your app didn't run out of space. To quote from https://stackoverflow.com/questions/7513185/what-are-reservedcodecachesize-and-initialcodecachesize https://stackoverflow.com/questions/7513185/what-are-reserve...: "Normally you'd not change this value. I think the default values are quite good balanced because this problems occur on very rare occasions only (in my experince)." Similarly, your organization tends to run the version of Java that the whole company supports. I have never worked anywhere where someone was allowed to use a different version than the rest of the company for performance reasons. In contrast, the advice to write low-level code for predictable, high performance has been virtually timeless and generally applicable across languages.
- hinkley 4y ago
  > the type of code executed on a page changes wildly depending on what you're doing and what actions you take. I can't speak for anyone else, but I feel like one of my goals for mature software is to start factoring out common bits of code so that a lot of the workloads either share code in common, or share output in common (avoiding code re-execution). There's a high correlation between Choose Your Own Adventure code that's doing random stuff for every request and my unhappiness on a project. I usually find at least a few highly effective people on any team who sympathize or agree. If at least 20% of the code isn't "in common" I'm uncomfortable. In common doesn't necessarily mean every task, but multiple tasks. EG, every task runs 5 of these 20 common concerns. I'm a sucker for complementary code and processes though. I am sure that some of my preferences in software design have an origin story in hardware sympathy, some in ergonomics/cognition, and a number in both. I couldn't begin to guess which opinions started with which observation at this point. Did I rationalize a human optimal solution or rationalize a hardware optimal one?
- fulafel 4y ago
  I think it depends on whether you are thinking about backend code or things like browsers. The results section of the article leads with: The research question in TCPT I am most interested in is RQ1 "Do Java microbenchmarks reach a steady state of performance?"
SethTro 4y ago
A good blog post has lots of hard parts (layout, scope, visuals, audience). Laurence Tratt, you nailed it for me! I loved the details you put in "Benchmarking methodology" and the clean layout with instructive visuals.
pizlonator 4y ago
For this research to be useful, the authors should switch to large benchmarks. VMs win in the average. Therefore, VMs have great cases, average cases, and crappy cases. The crappy cases will always exist, and that’s known to VM architects and it’s not a bug. Essentially this work is like criticizing a professional gambler for his biggest losses when the gambler is ahead in the average (or conversely praising him for his biggest wins when they’re behind in the average). Source: I build VMs for a living.
- stcredzero 4y ago
  The crappy cases will always exist, and that’s known to VM architects and it’s not a bug. I think this is very much the lens of the VM developer. From a different POV, why isn't it valid to ask, "Why can't this also be fast?" Why is that a feature and not a bug? Essentially this work is like criticizing a professional gambler for his biggest losses when the gambler is ahead in the average The losses are still losses. Maybe we should try to eliminate the "gambling?"
  - pizlonator 4y ago
    VMs are fundamentally about gambling. If you knew statically what to do in the compiler then you’d run an ahead of time (AOT) compiler. VMs are for those cases where an AOT would have been slower, and empirically, languages like JS, Java, and Lua perform much better with a dynamic VM than with an AOT and that’s why we use VMs. So, eliminating gambling is like saying we should just do AOTs. We could but then we’d be slower. > From a different POV, why isn't it valid to ask, "Why can't this also be fast?" Why is that a feature and not a bug? A professional gambler ought not ask this question. A professional gambler ought instead ask: how am I doing in the average? Hence the flaw in this research. It’s bad gambling.
    - stcredzero 4y ago
      A professional gambler ought not ask this question. A professional gambler ought instead ask: how am I doing in the average? Hence the flaw in this research. It’s bad gambling. I should think "not exactly." A professional gambler would take a look at each game, and determine if the expected payoff makes it worth playing. Here is where the analogy falls down. Not every analogue of "game" is the same! I worked for awhile for a company that was effectively a JIT VM vendor, and we found scenarios where our JIT could outperform things like a naively written YACC/LEX compiler. (Or a web templating engine. If we cranked up the size of the JIT code cache that ran at speeds expected of a C implementation.)
hinkley 4y ago
If I were the King of the Forest, our routing fabric would utilize something like nginx's ramping weight feature to throttle new servers for a few minutes before they reached full membership in the cluster. As things are we end up running something more like blue-green deployments and hit the dormant side with a stress testing tool. We haven't really come up with a better solution, though we have steadily reduced both the time necessary to warm up the servers and the worst case behavior if you accidentally skip the warming step. Today you will just have a very bad time. Originally circuits would blow like crazy.
hinkley 4y ago
It's interesting how often the same stories play out for different interpreter implementations. Everyone tries: - a multi-pass JIT - interpretting the input directly to reduce time to first op - making the fastest transpiler they can and skipping the interpreter All of these are addressing different constraints and affect each other. For instance, a cheap transpiler is only slightly slower than an interpreter loop, and allows you to move the threshold for the JIT farther to the right. If you can avoid trying to optimize things that you will only be slightly successful at, you can invest more of your CPU budget in deeper optimization on the hottest paths. You are also running on-stack replacement less often, and in fewer scenarios, which may mean you make different tradeoffs there as well.
adwn 4y ago
> Here's an example of a "good" benchmark from our dataset which starts slow and hits a steady state of peak performance Am I misinterpreting their graph? The difference between "slow" and "peak performance" seems to be a factor of about 1.005, so a whopping 0.5% improvement after warmup?
- dan-robertson 4y ago
  Perhaps that program was mostly doing some regexp thing and so it was easy for the jit to optimise a few function calls and a for loop correctly?
winrid 4y ago
Note that certain things can cause your code to be un-jitted. If you run into a NPE, for example, what actually happens is the compiled code segfaults. Adding null checks everywhere would be slow, so instead the JVM lets it blow up. When the jitted code segfaults that bytecode goes back into interpreted mode and has to be compiled again w/ updated heuristics. Still, covering cases like this is easier than writing C. :)
- winrid 4y ago
  Also what's missing in this article is a relative comparison. What if the testing machines have some kind of cron or process that's messing up the microbenchmark? I didn't see that comparison here.
  - dan-robertson 4y ago
    Did you miss the bits about krun or running many iterations? Though that’s mostly from the older paper rather than the subject of this article.
    - winrid 4y ago
      I guess I missed that.
1letterunixname 4y ago
The best warmup is to have a long-running VM without memory leaks but with long-term stats, PGO-directed JIT similar to HotSpot.

87 points by ltratt 4y ago

I appreciate that these issues definitely complicate research and benchmarking of VM performance, but I'm confused about this question: "If you're a normal user, the results suggest that you're often not getting the performance you expect". Is that true? Most VMs are not simple, predictable systems that would ever reach a "steady state"—they're dynamic systems that are constantly executing different parts of the codebase and taking wildly different codepaths at different times. V8 is a great example of this—the type of code executed on a page changes wildly depending on what you're doing and what actions you take. Why would we even want to optimize it for "reaching a steady state", when different parts of the codebase may be more or less useful at different times? It seems more important to me to work on optimizations that can allow us to deoptimize parts of the codebase that we don't think we will be useful again, to save on memory, even if it involves sacrificing this theoretical notion of a "steady state"

chrisseaton 4y ago
> Most VMs are not simple, predictable systems that would ever reach a "steady state"—they're dynamic systems that are constantly executing different parts of the codebase and taking wildly different codepaths at different times. V8 is a great example of this—the type of code executed on a page changes wildly depending on what you're doing and what actions you take. That is already the exact point being made by Laurie and Edd's research. > Why would we even want to optimize it for "reaching a steady state", when different parts of the codebase may be more or less useful at different times? A steady state for whatever the application is doing right now. Not a single, statically known steady state, without considering what it's doing. > It seems more important to me to work on optimizations that can allow us to deoptimize parts of the codebase that we don't think we will be useful again, to save on memory, even if it involves sacrificing this theoretical notion of a "steady state" I think that's a completely solved problem isn't it? You can GC a compiled method. What more do you want?
- dan-robertson 4y ago
  It smells like a caching problem rather than a gc problem. Surely the compiled method would be kept live by the pointer from its interpreted cousin or from its class’s vtable or whatever.
  - chrisseaton 4y ago
    It’s a kind of weak pointer.
    - dan-robertson 4y ago
      Yeah I see the comparison, it just feels… weak. With weak pointers there isn’t really much of a ‘gc keeps around objects that are used a lot’ thing going on whereas that fits much more nicely for a cache and for this situation.
hedora 4y ago
After thinking about these issues for over a decade, and having the rug repeatedly pulled out from under me by Java, I've come to the conclusion that the easiest path forward is to stick code snippets into godbolt, or to disassemble optimized binaries from ahead of time compiled languages. (I didn't say doing this is easy; just that it is easier than reasoning about a JIT)
- PathOfEclipse 4y ago
  What's the difference in practice between a JIT and an AOT compiler that supports optimizations based on runtime analysis (I forgot the technical term for that. Profile-guided optimization, maybe?)? I think the real issue is, the higher level the language, the harder it is to reason about what your code will actually get compiled down to. If you want predictable performance in Java, or any high-level language, in my experience you have to essentially write C code in that language, or at least get as close as possible. In Java, that usually means: * sticking to primitives or arrays of primitives as much as possible. The language has no support for user-defined value types! * Sticking to regular for loops, while loops, and other similar control structures. * Using offheap memory via Unsafe or similar API when feasible.
  - jeffbee 4y ago
    The things that most Java applications need to do to reduce the variance of runtime outcomes are sadly a little more basic: use a modern VM, set the compilation flags appropriately, and make sure the code caches are large enough to hold all your hot functions.
    - PathOfEclipse 4y ago
      My experience leads me to the exact opposite conclusions. For example, your code cache is either big enough or it's not. It's a pretty binary situation and a simple checkbox to ensure your app didn't run out of space. To quote from https://stackoverflow.com/questions/7513185/what-are-reservedcodecachesize-and-initialcodecachesize https://stackoverflow.com/questions/7513185/what-are-reserve...: "Normally you'd not change this value. I think the default values are quite good balanced because this problems occur on very rare occasions only (in my experince)." Similarly, your organization tends to run the version of Java that the whole company supports. I have never worked anywhere where someone was allowed to use a different version than the rest of the company for performance reasons. In contrast, the advice to write low-level code for predictable, high performance has been virtually timeless and generally applicable across languages.
hinkley 4y ago
> the type of code executed on a page changes wildly depending on what you're doing and what actions you take. I can't speak for anyone else, but I feel like one of my goals for mature software is to start factoring out common bits of code so that a lot of the workloads either share code in common, or share output in common (avoiding code re-execution). There's a high correlation between Choose Your Own Adventure code that's doing random stuff for every request and my unhappiness on a project. I usually find at least a few highly effective people on any team who sympathize or agree. If at least 20% of the code isn't "in common" I'm uncomfortable. In common doesn't necessarily mean every task, but multiple tasks. EG, every task runs 5 of these 20 common concerns. I'm a sucker for complementary code and processes though. I am sure that some of my preferences in software design have an origin story in hardware sympathy, some in ergonomics/cognition, and a number in both. I couldn't begin to guess which opinions started with which observation at this point. Did I rationalize a human optimal solution or rationalize a hardware optimal one?
fulafel 4y ago
I think it depends on whether you are thinking about backend code or things like browsers. The results section of the article leads with: The research question in TCPT I am most interested in is RQ1 "Do Java microbenchmarks reach a steady state of performance?"

SethTro 4y ago

A good blog post has lots of hard parts (layout, scope, visuals, audience). Laurence Tratt, you nailed it for me! I loved the details you put in "Benchmarking methodology" and the clean layout with instructive visuals.

pizlonator 4y ago

For this research to be useful, the authors should switch to large benchmarks. VMs win in the average. Therefore, VMs have great cases, average cases, and crappy cases. The crappy cases will always exist, and that’s known to VM architects and it’s not a bug. Essentially this work is like criticizing a professional gambler for his biggest losses when the gambler is ahead in the average (or conversely praising him for his biggest wins when they’re behind in the average). Source: I build VMs for a living.

stcredzero 4y ago
The crappy cases will always exist, and that’s known to VM architects and it’s not a bug. I think this is very much the lens of the VM developer. From a different POV, why isn't it valid to ask, "Why can't this also be fast?" Why is that a feature and not a bug? Essentially this work is like criticizing a professional gambler for his biggest losses when the gambler is ahead in the average The losses are still losses. Maybe we should try to eliminate the "gambling?"
- pizlonator 4y ago
  VMs are fundamentally about gambling. If you knew statically what to do in the compiler then you’d run an ahead of time (AOT) compiler. VMs are for those cases where an AOT would have been slower, and empirically, languages like JS, Java, and Lua perform much better with a dynamic VM than with an AOT and that’s why we use VMs. So, eliminating gambling is like saying we should just do AOTs. We could but then we’d be slower. > From a different POV, why isn't it valid to ask, "Why can't this also be fast?" Why is that a feature and not a bug? A professional gambler ought not ask this question. A professional gambler ought instead ask: how am I doing in the average? Hence the flaw in this research. It’s bad gambling.
  - stcredzero 4y ago
    A professional gambler ought not ask this question. A professional gambler ought instead ask: how am I doing in the average? Hence the flaw in this research. It’s bad gambling. I should think "not exactly." A professional gambler would take a look at each game, and determine if the expected payoff makes it worth playing. Here is where the analogy falls down. Not every analogue of "game" is the same! I worked for awhile for a company that was effectively a JIT VM vendor, and we found scenarios where our JIT could outperform things like a naively written YACC/LEX compiler. (Or a web templating engine. If we cranked up the size of the JIT code cache that ran at speeds expected of a C implementation.)

hinkley 4y ago

If I were the King of the Forest, our routing fabric would utilize something like nginx's ramping weight feature to throttle new servers for a few minutes before they reached full membership in the cluster. As things are we end up running something more like blue-green deployments and hit the dormant side with a stress testing tool. We haven't really come up with a better solution, though we have steadily reduced both the time necessary to warm up the servers and the worst case behavior if you accidentally skip the warming step. Today you will just have a very bad time. Originally circuits would blow like crazy.

hinkley 4y ago

It's interesting how often the same stories play out for different interpreter implementations. Everyone tries: - a multi-pass JIT - interpretting the input directly to reduce time to first op - making the fastest transpiler they can and skipping the interpreter All of these are addressing different constraints and affect each other. For instance, a cheap transpiler is only slightly slower than an interpreter loop, and allows you to move the threshold for the JIT farther to the right. If you can avoid trying to optimize things that you will only be slightly successful at, you can invest more of your CPU budget in deeper optimization on the hottest paths. You are also running on-stack replacement less often, and in fewer scenarios, which may mean you make different tradeoffs there as well.

adwn 4y ago

> Here's an example of a "good" benchmark from our dataset which starts slow and hits a steady state of peak performance Am I misinterpreting their graph? The difference between "slow" and "peak performance" seems to be a factor of about 1.005, so a whopping 0.5% improvement after warmup?

dan-robertson 4y ago
Perhaps that program was mostly doing some regexp thing and so it was easy for the jit to optimise a few function calls and a for loop correctly?

winrid 4y ago

Note that certain things can cause your code to be un-jitted. If you run into a NPE, for example, what actually happens is the compiled code segfaults. Adding null checks everywhere would be slow, so instead the JVM lets it blow up. When the jitted code segfaults that bytecode goes back into interpreted mode and has to be compiled again w/ updated heuristics. Still, covering cases like this is easier than writing C. :)

winrid 4y ago
Also what's missing in this article is a relative comparison. What if the testing machines have some kind of cron or process that's messing up the microbenchmark? I didn't see that comparison here.
- dan-robertson 4y ago
  Did you miss the bits about krun or running many iterations? Though that’s mostly from the older paper rather than the subject of this article.
  - winrid 4y ago
    I guess I missed that.

1letterunixname 4y ago

The best warmup is to have a long-running VM without memory leaks but with long-term stats, PGO-directed JIT similar to HotSpot.