4 ms·

JVM Performance Comparison for JDK 21

105 points by mfiguiere 3y ago

onetime5434 3y ago
"The Oracle GraalVM JIT compiler performs the same optimization as the Oracle GraalVM JIT compiler."
- danmur 3y ago
  whew lucky
- karussell 3y ago
  I think there is a CE somewhere missing :)
jt2190 3y ago
> Each benchmark uses 5x10s warm-up iterations… How do I read “5x10s”? 10^5?
- nicktelford 3y ago
  5 warm-up iterations of 10 seconds each.
traspler 3y ago
I never really understood why the Oracle/Enterprise version of GraalVM has better performance (also advertised) compared to the CE version. Is there any fundamental difference?
- deleted 3y ago
  [deleted]
- pjmlp 3y ago
  Yes, it is similar to C and C++ commercial compilers with different prices levels, written as modular compilers where the set of available optimization passes, among other features isn't available. For example, it used to be so that stuff like OpenMP was only available in professional licenses for C, C++ and Fortran compilers.
  - lbalazscs 3y ago
    "Oracle GraalVM" (the former "Enterprise Edition") is now free, but (unlike "GraalVM Community Edition") not open source.
    - maksut 3y ago
      It is free to try. But may not be free for commercial use. Some random blog post says check it with Oracle if you want to sell your products/services with it.
      - igouy 3y ago
        "The GFTC is the license for GraalVM for JDK 21, GraalVM for JDK 17, GraalVM for JDK 20 and later releases. Subject to the conditions of the license, it permits free use for all users – even commercial and production use." "What is the new “GraalVM Free Terms and Conditions (GFTC) including License for Early Adopter Versions” License?" https://www.oracle.com/java/technologies/javase/jdk-faqs.html https://www.oracle.com/java/technologies/javase/jdk-faqs.htm... Seems like there's a different license for GraalVM 22.
        banashark 3y ago
        Diffing the licenses shows that the only change was to add clarity around graalvm as opposed to a generic program: "For clarity, those portions of a Program included in otherwise unmodified software that is produced as output resulting from running the unmodified Program (such as may be produced by use of GraalVM Native Image) shall be deemed to be an unmodified Program for the purposes of this license." Other than that, it appears the same. The thing I'm wondering about is how this matches with everyone saying it's "free to use" when the following is in both versions of the license (underscore _emphasis_ mine): "(b) redistribute the unmodified Program and Program Documentation, under the terms of this License, provided that __You do not charge Your licensees any fees associated with such distribution or use of the Program__, including, without limitation, fees for products that include or are bundled with a copy of the Program or for services that involve the use of the distributed Program." This to me reads as if: - You cannot charge for licensing fees on graalvm built distributed pieces of software (licensing for a desktop application, cli program, game, etc) - You cannot charge for licensing fees on services that involve the use of graalvm built software (SaaS software that uses the graalvm built binary in a k8s container, lambad, etc as part of the service topology) My gut reaction (since this is Oracle) is that this was an ambiguity oversight in the language that is now being corrected to lock down this "free" usage of the premium version, but I may be overly pessimistic. Is anyone using this in prod based on it being "free" and has any info from their legal team or otherwise? Mostly looking to be corrected in my likely misinterpretation.
- speed_spread 3y ago
  Market segmentation to justify charging companies for the the compiler with more optimizations. If its not your money and can convince your boss you need it...
- casselc 3y ago
  The primary performance benefits with Enterprise are access to the G1 garbage collector and profile-guided optimization
  - dastbe 3y ago
    ? both of these things are available in openjdk distributions
    - maksut 3y ago
      Apparently those are not available for native executibles produced by graalvm community edition. Profile guided optimisation for native executibles sounds cool. Too bad Oracle wants to keep it closed for monetisation.
  - neonsunset 3y ago
    PGO (both static and dynamic) is something that is offered to you in the FOSS form in .NET land.
    - pjmlp 3y ago
      Meanwhile .NET land requires writing reflection free specific code for AOT, while GraalVM can handle reflection just fine. ART also does PGO for free, as does OpenJ9. Both of which also handle reflection. The thing with Java ecosystem is choice of implementations, instead of Microsoft's single implementation, specially now since most of Xamarin's work has been consumed into modern .NET.
      - neonsunset 3y ago
        This is simply not true. AOT requires code that is free of uses of JIT-specific features like runtime reflection-emit or assembly loading, which is not possible without build instrumentation for AOT implementations on other platforms either[0], because it by definition needs a JIT compiler. Reflection itself not only works but is sometimes faster because closed world guarantee by AOT allows for a more streamlined implementation. With that said, trimming, also known as tree-shaking, which is required to not have massive binary size, has to know at compile-time which types, members and metadata tokens are reachable. This means that certain patterns of unbound reflection are by definition not statically analyzable and require the user to annotate (with [RequiresUnreferencedCode(params)]) the corresponding types with details of what they need the trimmer/linker to preserve (the entire assembly can also be rooted but, well, that's bad for size). Other features like compiled expressions have received proper interpreter fallback which makes them just work(tm) without developer intervention. You can also see that GraalVM Native Image is subject to very similar restrictions[1]. [0] Like pre-generating generic types/method bodies for Unity with special tooling. NativeAOT currently has (planned to be deprecated) rd.xml to manually instantiate certain generic methods and root specific types, replace their bodies, etc., it will probably be fully superseded by further expansion of attributes-style annotations API or obsoleted by improving compile-time analysis in the future. [1] https://www.graalvm.org/22.1/reference-manual/native-image/Limitations/ https://www.graalvm.org/22.1/reference-manual/native-image/L... p.s.: quick search shows that Java Swing is plagued by similar issues with Native Image (not mentioning having to generate metadata annotations) to making .NET's GUI frameworks work with NativeAOT save for certain blessed templates/workloads introduced in .NET 8. Java AWT appears to be faring somewhat better on hello world examples but I suspect it experience the same fate as the templates for WinForms that the community has come up with so far (let me know if that's not the case).
berlinbrowngalt 3y ago
Java, still not debt.
imoverclocked 3y ago
TL;DR: real-world performance between the three will be highly dependent on just how bad your code is in different metrics. Also: the CE version shouldn’t be considered representative of the non-CE version performance.
rsprinkle 3y ago
It would be useful to see memory utilization while performing the benchmarks. Historically GraalVM used ALOT of memory.
kjellsbells 3y ago
I often wonder if there should be a suite of tests that deliberately dont use well-tuned algorithms, or optimal ones, as the basis for benchmarks. In the real world, people dont choose optimal paths in code very often. Or what was once optimal becomes suboptimal because the context changes. Supposing that a benchmark chose to do, say, bubblesort on an insanely large list. Would that lead to more insight around how the compiler would act in the real world? Or are the pitfalls so well understood that they are universally optimized away by the compiler's earliest stages or processing? If I were selling, say, Azul or some other alternative JDK, I might make a point of noting that it does a superior job of dealing with the code as it exists today, rather than as one might wish it to be.
- codr7 3y ago
  I like to use naive recursive fibonacci to bootstrap performance testing of my interpreters. It's a pretty lousy algorithm, but it does a good job of stress testing instruction dispatch and function calls.
- hocuspocus 3y ago
  Anecdotally I've seen much bigger gains on my team's Spark jobs that use complex business logic (i.e. our own hand rolled algorithms in idiomatic Scala) compared to plain ETL jobs that use the vanilla Spark API, which is now very mature and optimized. GraalVM does a really impressive job with mediocre code :-)
projectileboy 3y ago
This is great work, although benchmarks that do not explicitly include analyses of each JVM’s GC options and their behaviors don’t help me much in practice. I’m not doing scientific computing; I need to know what kind of GC pauses I get and how often.
- freedom-fries 3y ago
  GC pauses are dependent on the code and its workload so general benchmarking is rarely useful. You can have a code that works really nicely with GC at a given throughput and request pattern, but not at all at a different load profile. Minor implementation change can also dramatically alter the GC workload. A generic work-queue, where one thread is enqueuing tasks (allocating memory) and another is consuming them and deallocating memory can make it hard for GC to do its work, while minor changes so that queue is fixed-sized and does not allocate can dramatically change the GC workload. Even really simple optimizations where the `enqueue` may just copy to its queue-owned buffers can make allocations happen exclusively from TLAB with bump allocation which are essentially no-ops for GC & the allocator. Generic benchmarking happens but is more the domain of GC developers, and not very useful for developers – because usually, your workload is different from mine.
codr7 3y ago
The Oracle GraalVM JIT compiler performs the same optimization as the Oracle GraalVM JIT compiler. Makes sense.
- stolsvik 3y ago
  You did notice the "CE" in one of them? Community Edition. Evidently a dumbed-down variant so that there is some incentive in buying the better one. I have they give up on that at some point.
  - codr7 3y ago
    No I did not, there is no CE to be found in that sentence and I grabbed straight off the page.

105 points by mfiguiere 3y ago

onetime5434 3y ago

"The Oracle GraalVM JIT compiler performs the same optimization as the Oracle GraalVM JIT compiler."

danmur 3y ago
whew lucky
karussell 3y ago
I think there is a CE somewhere missing :)

jt2190 3y ago

> Each benchmark uses 5x10s warm-up iterations… How do I read “5x10s”? 10^5?

nicktelford 3y ago
5 warm-up iterations of 10 seconds each.

traspler 3y ago

I never really understood why the Oracle/Enterprise version of GraalVM has better performance (also advertised) compared to the CE version. Is there any fundamental difference?

deleted 3y ago
[deleted]
pjmlp 3y ago
Yes, it is similar to C and C++ commercial compilers with different prices levels, written as modular compilers where the set of available optimization passes, among other features isn't available. For example, it used to be so that stuff like OpenMP was only available in professional licenses for C, C++ and Fortran compilers.
- lbalazscs 3y ago
  "Oracle GraalVM" (the former "Enterprise Edition") is now free, but (unlike "GraalVM Community Edition") not open source.
  - maksut 3y ago
    It is free to try. But may not be free for commercial use. Some random blog post says check it with Oracle if you want to sell your products/services with it.
    - igouy 3y ago
      "The GFTC is the license for GraalVM for JDK 21, GraalVM for JDK 17, GraalVM for JDK 20 and later releases. Subject to the conditions of the license, it permits free use for all users – even commercial and production use." "What is the new “GraalVM Free Terms and Conditions (GFTC) including License for Early Adopter Versions” License?" https://www.oracle.com/java/technologies/javase/jdk-faqs.html https://www.oracle.com/java/technologies/javase/jdk-faqs.htm... Seems like there's a different license for GraalVM 22.
      - banashark 3y ago
        Diffing the licenses shows that the only change was to add clarity around graalvm as opposed to a generic program: "For clarity, those portions of a Program included in otherwise unmodified software that is produced as output resulting from running the unmodified Program (such as may be produced by use of GraalVM Native Image) shall be deemed to be an unmodified Program for the purposes of this license." Other than that, it appears the same. The thing I'm wondering about is how this matches with everyone saying it's "free to use" when the following is in both versions of the license (underscore _emphasis_ mine): "(b) redistribute the unmodified Program and Program Documentation, under the terms of this License, provided that __You do not charge Your licensees any fees associated with such distribution or use of the Program__, including, without limitation, fees for products that include or are bundled with a copy of the Program or for services that involve the use of the distributed Program." This to me reads as if: - You cannot charge for licensing fees on graalvm built distributed pieces of software (licensing for a desktop application, cli program, game, etc) - You cannot charge for licensing fees on services that involve the use of graalvm built software (SaaS software that uses the graalvm built binary in a k8s container, lambad, etc as part of the service topology) My gut reaction (since this is Oracle) is that this was an ambiguity oversight in the language that is now being corrected to lock down this "free" usage of the premium version, but I may be overly pessimistic. Is anyone using this in prod based on it being "free" and has any info from their legal team or otherwise? Mostly looking to be corrected in my likely misinterpretation.
speed_spread 3y ago
Market segmentation to justify charging companies for the the compiler with more optimizations. If its not your money and can convince your boss you need it...
casselc 3y ago
The primary performance benefits with Enterprise are access to the G1 garbage collector and profile-guided optimization
- dastbe 3y ago
  ? both of these things are available in openjdk distributions
  - maksut 3y ago
    Apparently those are not available for native executibles produced by graalvm community edition. Profile guided optimisation for native executibles sounds cool. Too bad Oracle wants to keep it closed for monetisation.
- neonsunset 3y ago
  PGO (both static and dynamic) is something that is offered to you in the FOSS form in .NET land.
  - pjmlp 3y ago
    Meanwhile .NET land requires writing reflection free specific code for AOT, while GraalVM can handle reflection just fine. ART also does PGO for free, as does OpenJ9. Both of which also handle reflection. The thing with Java ecosystem is choice of implementations, instead of Microsoft's single implementation, specially now since most of Xamarin's work has been consumed into modern .NET.
    - neonsunset 3y ago
      This is simply not true. AOT requires code that is free of uses of JIT-specific features like runtime reflection-emit or assembly loading, which is not possible without build instrumentation for AOT implementations on other platforms either[0], because it by definition needs a JIT compiler. Reflection itself not only works but is sometimes faster because closed world guarantee by AOT allows for a more streamlined implementation. With that said, trimming, also known as tree-shaking, which is required to not have massive binary size, has to know at compile-time which types, members and metadata tokens are reachable. This means that certain patterns of unbound reflection are by definition not statically analyzable and require the user to annotate (with [RequiresUnreferencedCode(params)]) the corresponding types with details of what they need the trimmer/linker to preserve (the entire assembly can also be rooted but, well, that's bad for size). Other features like compiled expressions have received proper interpreter fallback which makes them just work(tm) without developer intervention. You can also see that GraalVM Native Image is subject to very similar restrictions[1]. [0] Like pre-generating generic types/method bodies for Unity with special tooling. NativeAOT currently has (planned to be deprecated) rd.xml to manually instantiate certain generic methods and root specific types, replace their bodies, etc., it will probably be fully superseded by further expansion of attributes-style annotations API or obsoleted by improving compile-time analysis in the future. [1] https://www.graalvm.org/22.1/reference-manual/native-image/Limitations/ https://www.graalvm.org/22.1/reference-manual/native-image/L... p.s.: quick search shows that Java Swing is plagued by similar issues with Native Image (not mentioning having to generate metadata annotations) to making .NET's GUI frameworks work with NativeAOT save for certain blessed templates/workloads introduced in .NET 8. Java AWT appears to be faring somewhat better on hello world examples but I suspect it experience the same fate as the templates for WinForms that the community has come up with so far (let me know if that's not the case).

berlinbrowngalt 3y ago

Java, still not debt.

imoverclocked 3y ago

TL;DR: real-world performance between the three will be highly dependent on just how bad your code is in different metrics. Also: the CE version shouldn’t be considered representative of the non-CE version performance.

rsprinkle 3y ago

It would be useful to see memory utilization while performing the benchmarks. Historically GraalVM used ALOT of memory.

kjellsbells 3y ago

I often wonder if there should be a suite of tests that deliberately dont use well-tuned algorithms, or optimal ones, as the basis for benchmarks. In the real world, people dont choose optimal paths in code very often. Or what was once optimal becomes suboptimal because the context changes. Supposing that a benchmark chose to do, say, bubblesort on an insanely large list. Would that lead to more insight around how the compiler would act in the real world? Or are the pitfalls so well understood that they are universally optimized away by the compiler's earliest stages or processing? If I were selling, say, Azul or some other alternative JDK, I might make a point of noting that it does a superior job of dealing with the code as it exists today, rather than as one might wish it to be.

codr7 3y ago
I like to use naive recursive fibonacci to bootstrap performance testing of my interpreters. It's a pretty lousy algorithm, but it does a good job of stress testing instruction dispatch and function calls.
hocuspocus 3y ago
Anecdotally I've seen much bigger gains on my team's Spark jobs that use complex business logic (i.e. our own hand rolled algorithms in idiomatic Scala) compared to plain ETL jobs that use the vanilla Spark API, which is now very mature and optimized. GraalVM does a really impressive job with mediocre code :-)

projectileboy 3y ago

This is great work, although benchmarks that do not explicitly include analyses of each JVM’s GC options and their behaviors don’t help me much in practice. I’m not doing scientific computing; I need to know what kind of GC pauses I get and how often.

freedom-fries 3y ago
GC pauses are dependent on the code and its workload so general benchmarking is rarely useful. You can have a code that works really nicely with GC at a given throughput and request pattern, but not at all at a different load profile. Minor implementation change can also dramatically alter the GC workload. A generic work-queue, where one thread is enqueuing tasks (allocating memory) and another is consuming them and deallocating memory can make it hard for GC to do its work, while minor changes so that queue is fixed-sized and does not allocate can dramatically change the GC workload. Even really simple optimizations where the `enqueue` may just copy to its queue-owned buffers can make allocations happen exclusively from TLAB with bump allocation which are essentially no-ops for GC & the allocator. Generic benchmarking happens but is more the domain of GC developers, and not very useful for developers – because usually, your workload is different from mine.

codr7 3y ago

The Oracle GraalVM JIT compiler performs the same optimization as the Oracle GraalVM JIT compiler. Makes sense.

stolsvik 3y ago
You did notice the "CE" in one of them? Community Edition. Evidently a dumbed-down variant so that there is some incentive in buying the better one. I have they give up on that at some point.
- codr7 3y ago
  No I did not, there is no CE to be found in that sentence and I grabbed straight off the page.