3 ms·

Yeah, I would like to see them to, especially when I see something like this: > When we benchmarked JDK 15, we saw that Java 15 was 11.24% faster than Java 11.

Yeah, I would like to see them to, especially when I see something like this: > When we benchmarked JDK 15, we saw that Java 15 was 11.24% faster than Java 11. Now, the gain of Java 17 over Java 11 is less. Does that mean that Java 17 is slower than Java 15? > Well, no. Java 17 is faster than Java 15 too. Those previous benchmarks were run on a different codebase (OptaPlanner 7.44 instead of 8.10). Don’t compare apples and oranges. Author is defending their other article by discrediting their own methods - what those percent values even mean if they differ so much between different versions of the same benchmark? Shouldn't benchmarks be profiled for specific workload types to simulate real life examples? How could you possibly change the workload so much it affects result so significantly and not change the nature of your benchmark?

ge0ffrey 5y ago
Good questions (I am the author). It's complex though. 1) They are not the same benchmark. The Java 15 blog used optaplanner 7.x with optaplanner-examples 7.x, which use scoreDRL. The Java 17 blog used 8.x versions, which use ConstraintStreams. That scoreDRL vs CS is a huge difference, which you see on the numbers (run on the same machine). 2) Remove the Machine Reassignment (B1 and B10) numbers, and it will look consistent: Java 17 is always better. The real question is why is every JDK's (11 too!) performance is predictable on most use cases, but horribly so on the Machine Reassignement case? 3) I need to share the numbers of the 3 raw runs - and probably do more runs - to clearly show and explain what's going on there. It looks to damn fishy.