5 ms·

The "Spark CPU usage with GraalVM" chart is ... not zero-based. Using that kind of chart to show 10% reduction isn't an honest way to show the improvement. Use

The "Spark CPU usage with GraalVM" chart is ... not zero-based. Using that kind of chart to show 10% reduction isn't an honest way to show the improvement. Use a zero-based chart, and show a longer time period. Then the gap is clear, even though not as dramatic. Also, the article probably shouldn't talk about the general benchmark, and probably better to omit the performance of the Enterprise edition, because Facebook is probably not using it. (It's not clear from the article which version of GraalVM FB uses, but I don't think they pay for it.) We all know 10% CPU reduction is superb, especially on the Facebook scale. And we also know GraalVM is great technology. So please don't try to fool the readers. By the way, it also mentions that they moved Presto to GraalVM and saw an improvement. It can be as significant, and would be nice to show the number if there is a latency improvement because Presto is interactive tool and the improvement can end up productivity gain, in addition to saving the CPU cycles.

dariusj18 5y ago
> better to omit the performance of the Enterprise edition, because Facebook is probably not using it unless the point of this post is to attempt to sell the Enterprise edition to FB
- the_arun 5y ago
  Looks like it is a sales pitch for Enterprise Edition
  - native_samples 5y ago
    Well it's on the GraalVM blog and GraalVM has two editions. Facebook probably don't care how many copies of EE Oracle sell, but it seems pretty reasonable for Oracle to use the adoption of GraalVM tech by large corporations to cross-sell EE especially because the justification for adopting it is performance, and EE yields much better performance gains again.
amarshall 5y ago
> Use a zero-based chart Agreed, but that visualization is even worse because it lacks a y-axis scale entirely.
- disgruntledphd2 5y ago
  One of the things I learned as a megacorp data scientist is that you should always remove the scales on the y-axis in anything vaguely public, because of the risk of material information leak. (Think about user growth as a particularly good example of a variable like this).
  - amarshall 5y ago
    If absolute values are a concern then why not replace the unit with % change? One could, assuming linear scale, reverse engineer that onto this plot anyway. Some plots, of course, might stand better on their own without a y-axis scale (histogram distribution probably). To be honest, this plot is pretty useless either way; who cares about the trend it took downward since we have little or no context on what happened at any given point?
vips7L 5y ago
It's honestly pretty amazing that graal enterprise can offer 42% performance over HotSpot/C2. You would think Oracle would want to upstream those improvements to HotSpot to keep it as fast as possible.
- mey 5y ago
  This is the product they are selling. Unfortunately, this approach is what's keeping me away from Graal in general as well.
- mumblemumble 5y ago
  For one specific use case. Given that we're talking unpsecified Apache Spark workloads, it's a little hard to know exactly what's going on. I think that the author is correct, though, to point out that there's reason to believe that Spark workloads might see a larger benefit than other kinds of Java applications.
- geodel 5y ago
  > You would think Oracle would want to upstream ... Why would I think this and not that Oracle would want to make money off their unique IP?
- native_samples 5y ago
  The Graal team is huge. Like, I don't know, at least 70 people and probably more. So that's very expensive. Oracle Java is best understood as two competing teams at this point. In theory GraalVM started out as a research project to feed improvements back to HotSpot and some of that they've done, like with the JVMCI interface that was added. But it didn't work out that way for whatever reason and GraalVM is now effectively a competitor to the 'classical' Sun Java product guys. They have different brand names, commercial models, feature priorities, price points, they are different organisations within the company. Fortunately GraalVM is not a fork of OpenJDK - it's a bunch of plugins and extra components, more or less. But at some level the difference is semantic because if you swap out the whole JIT compiler then maybe it actually is a fork? Not sure. Anyway the original research vision was that C2 would be replaced by Graal indeed. The politics of merging such a total rewrite would be difficult at the best of times and it seems Oracle hasn't managed it, or perhaps they like having some internal competition for the OpenJDK guys. Certainly Graal team cares a lot more about modern marketing, about getting their tech adopted into other Oracle products and things that presumably Oracle management care about. Like the graalvm.org website vs the OpenJDK website is no competition. The former feels like an actual product, the latter is a bunch of wiki pages more or less.
nojito 5y ago
You don’t need zeros on y axis for line charts. The information you are trying to convey is the variability of the change over time.
- toomanyducks 5y ago
  right, but what variability? how much? what time period? without labels and numbers, the graph is pretty meaningless in that department. All I know is that something changed, which is not enough information to be at all useful.
hinkley 5y ago
Conversely, the first chart understates the effect. 0% improvement (1.0) is 5/8th of the length of the Y axis, making a 42% improvement look barely interesting. Probably should have opted for a different chart here.

dariusj18 5y ago

> better to omit the performance of the Enterprise edition, because Facebook is probably not using it unless the point of this post is to attempt to sell the Enterprise edition to FB

the_arun 5y ago
Looks like it is a sales pitch for Enterprise Edition
- native_samples 5y ago
  Well it's on the GraalVM blog and GraalVM has two editions. Facebook probably don't care how many copies of EE Oracle sell, but it seems pretty reasonable for Oracle to use the adoption of GraalVM tech by large corporations to cross-sell EE especially because the justification for adopting it is performance, and EE yields much better performance gains again.

amarshall 5y ago

> Use a zero-based chart Agreed, but that visualization is even worse because it lacks a y-axis scale entirely.

disgruntledphd2 5y ago
One of the things I learned as a megacorp data scientist is that you should always remove the scales on the y-axis in anything vaguely public, because of the risk of material information leak. (Think about user growth as a particularly good example of a variable like this).
- amarshall 5y ago
  If absolute values are a concern then why not replace the unit with % change? One could, assuming linear scale, reverse engineer that onto this plot anyway. Some plots, of course, might stand better on their own without a y-axis scale (histogram distribution probably). To be honest, this plot is pretty useless either way; who cares about the trend it took downward since we have little or no context on what happened at any given point?

vips7L 5y ago

It's honestly pretty amazing that graal enterprise can offer 42% performance over HotSpot/C2. You would think Oracle would want to upstream those improvements to HotSpot to keep it as fast as possible.

mey 5y ago
This is the product they are selling. Unfortunately, this approach is what's keeping me away from Graal in general as well.
mumblemumble 5y ago
For one specific use case. Given that we're talking unpsecified Apache Spark workloads, it's a little hard to know exactly what's going on. I think that the author is correct, though, to point out that there's reason to believe that Spark workloads might see a larger benefit than other kinds of Java applications.
geodel 5y ago
> You would think Oracle would want to upstream ... Why would I think this and not that Oracle would want to make money off their unique IP?
native_samples 5y ago
The Graal team is huge. Like, I don't know, at least 70 people and probably more. So that's very expensive. Oracle Java is best understood as two competing teams at this point. In theory GraalVM started out as a research project to feed improvements back to HotSpot and some of that they've done, like with the JVMCI interface that was added. But it didn't work out that way for whatever reason and GraalVM is now effectively a competitor to the 'classical' Sun Java product guys. They have different brand names, commercial models, feature priorities, price points, they are different organisations within the company. Fortunately GraalVM is not a fork of OpenJDK - it's a bunch of plugins and extra components, more or less. But at some level the difference is semantic because if you swap out the whole JIT compiler then maybe it actually is a fork? Not sure. Anyway the original research vision was that C2 would be replaced by Graal indeed. The politics of merging such a total rewrite would be difficult at the best of times and it seems Oracle hasn't managed it, or perhaps they like having some internal competition for the OpenJDK guys. Certainly Graal team cares a lot more about modern marketing, about getting their tech adopted into other Oracle products and things that presumably Oracle management care about. Like the graalvm.org website vs the OpenJDK website is no competition. The former feels like an actual product, the latter is a bunch of wiki pages more or less.

nojito 5y ago

You don’t need zeros on y axis for line charts. The information you are trying to convey is the variability of the change over time.

toomanyducks 5y ago
right, but what variability? how much? what time period? without labels and numbers, the graph is pretty meaningless in that department. All I know is that something changed, which is not enough information to be at all useful.

hinkley 5y ago

Conversely, the first chart understates the effect. 0% improvement (1.0) is 5/8th of the length of the Y axis, making a 42% improvement look barely interesting. Probably should have opted for a different chart here.