3 ms·

> For example, C2 might behave differently than Graal and who knows what the future brings. Yes, the GraalVM compiler is more aggressive on some optimizations

> For example, C2 might behave differently than Graal and who knows what the future brings. Yes, the GraalVM compiler is more aggressive on some optimizations that can be especially helpful with streams. Here are numbers from my machine (using ops/s, so higher is better!): C2 (on Java(TM) SE Runtime Environment (build 1.8.0_251-b08)): Benchmark (warmup) Mode Cnt Score Error Units MyBenchmark.collectionsDeadCode true thrpt 10 2.179 ± 0.070 ops/s MyBenchmark.collectionsDeadCode false thrpt 10 1.640 ± 0.055 ops/s MyBenchmark.coollectionsBlackhole true thrpt 10 1.661 ± 0.109 ops/s MyBenchmark.coollectionsBlackhole false thrpt 10 1.571 ± 0.062 ops/s MyBenchmark.loopBlackhole true thrpt 10 1.829 ± 0.077 ops/s MyBenchmark.loopBlackhole false thrpt 10 1.855 ± 0.062 ops/s MyBenchmark.loopBlackholeOnly true thrpt 10 19.775 ± 0.184 ops/s MyBenchmark.loopBlackholeOnly false thrpt 10 20.213 ± 0.491 ops/s MyBenchmark.loopDeadCode true thrpt 10 2.022 ± 0.057 ops/s MyBenchmark.loopDeadCode false thrpt 10 1.990 ± 0.074 ops/s GraalVM Community Edition (current master, close to yesterday's 20.1 release, JDK 8): Benchmark (warmup) Mode Cnt Score Error Units MyBenchmark.collectionsDeadCode true thrpt 10 3.151 ± 0.118 ops/s MyBenchmark.collectionsDeadCode false thrpt 10 2.790 ± 0.053 ops/s MyBenchmark.coollectionsBlackhole true thrpt 10 2.713 ± 0.137 ops/s MyBenchmark.coollectionsBlackhole false thrpt 10 2.488 ± 0.031 ops/s MyBenchmark.loopBlackhole true thrpt 10 2.862 ± 0.073 ops/s MyBenchmark.loopBlackhole false thrpt 10 2.835 ± 0.043 ops/s MyBenchmark.loopBlackholeOnly true thrpt 10 24.396 ± 1.278 ops/s MyBenchmark.loopBlackholeOnly false thrpt 10 24.103 ± 0.957 ops/s MyBenchmark.loopDeadCode true thrpt 10 3.235 ± 0.097 ops/s MyBenchmark.loopDeadCode false thrpt 10 3.205 ± 0.168 ops/s GraalVM Enterprise Edition (current master, close to 20.1, JDK 8): Benchmark (warmup) Mode Cnt Score Error Units MyBenchmark.collectionsDeadCode true thrpt 10 3.748 ± 0.046 ops/s MyBenchmark.collectionsDeadCode false thrpt 10 2.879 ± 0.042 ops/s MyBenchmark.coollectionsBlackhole true thrpt 10 2.792 ± 0.059 ops/s MyBenchmark.coollectionsBlackhole false thrpt 10 2.479 ± 0.040 ops/s MyBenchmark.loopBlackhole true thrpt 10 3.070 ± 0.046 ops/s MyBenchmark.loopBlackhole false thrpt 10 3.077 ± 0.057 ops/s MyBenchmark.loopBlackholeOnly true thrpt 10 25.200 ± 0.760 ops/s MyBenchmark.loopBlackholeOnly false thrpt 10 25.752 ± 0.481 ops/s MyBenchmark.loopDeadCode true thrpt 10 3.812 ± 0.161 ops/s MyBenchmark.loopDeadCode false thrpt 10 3.909 ± 0.109 ops/s Amusingly, on the OP's benchmark GraalVM also does better than C2, and it somehow (I haven't tried to analyze this yet) makes the serial stream version as fast as the parallel one: C2: Benchmark Mode Cnt Score Error Units MyBenchmark.originalParallelStream thrpt 10 2.547 ± 0.336 ops/s MyBenchmark.originalSerialStream thrpt 10 1.719 ± 0.041 ops/s GraalVM Community Edition: Benchmark Mode Cnt Score Error Units MyBenchmark.originalParallelStream thrpt 10 2.809 ± 0.141 ops/s MyBenchmark.originalSerialStream thrpt 10 2.868 ± 0.090 ops/s (I work in the GraalVM compiler team at Oracle Labs, I don't speak for Oracle, I'm not claiming anything about whether this is a good or bad benchmark, etc.)

palinkapika 6y ago
The results for the parallel code are indeed interesting. However, the speedup < 2 with C2 is suspicious. Even on an old dual core / 4 thread system (common worker pool should default to 3 workers), I would expect it to be above 2 for this kind of workload. The last time I looked into Graal (over a year ago), I ran into something I could not make sense of. We were designing an API with methods similar to Arrays.setAll(double[], IntToDoubleFunction) which we expected to be called with large input arrays from many places in the code base. The performance of the code generated by C2 dropped as one would expect once you were using more that two functions (lambda functions were no longer inlined and invoked for each element). For simple arithmetic operations the runtime increased by around 5x. The performance of the code generated by Graal was competitive with C2's code for mono- and bimorphic scenarios for a much larger number of input functions. I don't recall the exact number, but I believe it was around 25. However, once we hit that threshold, performance tanked. I believe the difference was closer to 15x than than C2's 5x. We never figured out what caused this. Do you have an idea? Or, since this is probably outdated data, do you know what the expected behavior is nowadays? Is there maybe any documentation on this?
- gergo_barany 6y ago
  > However, the speedup < 2 with C2 is suspicious. Even on an old dual core / 4 thread system (common worker pool should default to 3 workers), I would expect it to be above 2 for this kind of workload. The speedup I found on C2 is 2.547 / 1.719 ~= 1.5, the OP's speedup (also on C2) was 0.71 s / 0.38 s ~= 1.9. Not the same factor, but not above 2 either. The benchmark is quite small, so the overhead of setting up the parallel computation might matter. The JDK version might matter as well. > Graal was competitive with C2's code for mono- and bimorphic scenarios for a much larger number of input functions. I don't recall the exact number, but I believe it was around 25. If we are talking about a call like Arrays.setAll(array, function) and you have one or two candidates for function, then you are mono- or bimorphic. If you have 25 candidates for function, you are highly polymorphic. I don't understand what you mean by a mono- or bimorphic scenario with up to 25 candidates. Performance will usually decrease as you transition from a call site with few candidates to one with many candidates. I agree that a factor of 15x looks steep, but it's hard to say more without context. The inliners (Community and Enterprise use different ones) are constantly being tweaked, though that's not my area of expertise. It's very possible that this behaves differently than it did more than a year ago. If you have a concrete JMH benchmark you can link to and are willing to post to the GraalVM Slack (see signup link from https://www.graalvm.org/community/ https://www.graalvm.org/community/), that would be great. Alternatively, you could drop me a line at <first . last at oracle dot com>, and I could have a look at a benchmark, without promising anything.

palinkapika 6y ago

The results for the parallel code are indeed interesting. However, the speedup < 2 with C2 is suspicious. Even on an old dual core / 4 thread system (common worker pool should default to 3 workers), I would expect it to be above 2 for this kind of workload. The last time I looked into Graal (over a year ago), I ran into something I could not make sense of. We were designing an API with methods similar to Arrays.setAll(double[], IntToDoubleFunction) which we expected to be called with large input arrays from many places in the code base. The performance of the code generated by C2 dropped as one would expect once you were using more that two functions (lambda functions were no longer inlined and invoked for each element). For simple arithmetic operations the runtime increased by around 5x. The performance of the code generated by Graal was competitive with C2's code for mono- and bimorphic scenarios for a much larger number of input functions. I don't recall the exact number, but I believe it was around 25. However, once we hit that threshold, performance tanked. I believe the difference was closer to 15x than than C2's 5x. We never figured out what caused this. Do you have an idea? Or, since this is probably outdated data, do you know what the expected behavior is nowadays? Is there maybe any documentation on this?

gergo_barany 6y ago
> However, the speedup < 2 with C2 is suspicious. Even on an old dual core / 4 thread system (common worker pool should default to 3 workers), I would expect it to be above 2 for this kind of workload. The speedup I found on C2 is 2.547 / 1.719 ~= 1.5, the OP's speedup (also on C2) was 0.71 s / 0.38 s ~= 1.9. Not the same factor, but not above 2 either. The benchmark is quite small, so the overhead of setting up the parallel computation might matter. The JDK version might matter as well. > Graal was competitive with C2's code for mono- and bimorphic scenarios for a much larger number of input functions. I don't recall the exact number, but I believe it was around 25. If we are talking about a call like Arrays.setAll(array, function) and you have one or two candidates for function, then you are mono- or bimorphic. If you have 25 candidates for function, you are highly polymorphic. I don't understand what you mean by a mono- or bimorphic scenario with up to 25 candidates. Performance will usually decrease as you transition from a call site with few candidates to one with many candidates. I agree that a factor of 15x looks steep, but it's hard to say more without context. The inliners (Community and Enterprise use different ones) are constantly being tweaked, though that's not my area of expertise. It's very possible that this behaves differently than it did more than a year ago. If you have a concrete JMH benchmark you can link to and are willing to post to the GraalVM Slack (see signup link from https://www.graalvm.org/community/ https://www.graalvm.org/community/), that would be great. Alternatively, you could drop me a line at <first . last at oracle dot com>, and I could have a look at a benchmark, without promising anything.