4 ms·
Benchmarks game doesn't do JIT warmup, which is extremely frustrating. The results are useless for long running server applications. It essentially benchmarks w
by eggsnbacon1 6y ago
Benchmarks game doesn't do JIT warmup, which is extremely frustrating. The results are useless for long running server applications. It essentially benchmarks which VM can compile fairly fast code quickly, not which one produces the fastest code.
Long running VM's like Hotspot in "server" mode do a lot of expensive optimizations over time. Benchmarks game makes no attempt to warm up these VM's, so it doesn't actually measure how fast a VM is in practical hosting scenarios.
Anecdotally, I have heard that Java is still faster than C# in most benchmarks when both VM's have had time to do a full JIT. I've also heard that Golang is much slower than C# and Java when their JITs are allowed to fully warm.
I've been meaning to build a "warmed up" version of benchmarks game specifically for testing VM languages but never got around to it. If someone else wants to pick up the torch I would be eternally grateful!
- igouy 6y agoYou haven't done a "warmed up" version of benchmarks game — so you don't know how much or how little difference it would make for those tiny tiny programs. https://benchmarksgame-team.pages.debian.net/benchmarksgame/sometimes-people-just-make-up-stuff.html#jvm-startup-time https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
- eggsnbacon1 6y agoFor fast languages like C# and Java many benchmarks are finished in a few seconds. By their tests, the warmup adds ~.3 seconds to these tiny programs. That's a long time, 10-20% in some benchmarks! In a couple benchmarks Java would probably be faster than C++ if the JVM was allowed to warm up
- igouy 6y ago> In a couple benchmarks Java would probably be faster… Please be specific. JMH timing for that spectral-norm program was 0.175s to 0.283s faster than the 4.29s elapsed time (16s cpu time). That's not 10-20% it's 6.6%. At best that might put the best Java spectral-norm program a little faster than the best Haskell program. At best that might mean the best Java spectral-norm program took 2x (twice as long as) the best C++ program. https://benchmarksgame-team.pages.debian.net/benchmarksgame/q6600/performance/spectralnorm.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... > … would probably be faster… Please take those tiny tiny programs and JMH and make your own measurements — you might believe measurements you make yourself.
- igouy 6y agoUpdated for java 14.0.1: "Let's compare our fastest-of-6 no warmup measurements against the fastest-of-25 (or 55 or 175) with warmup JMH SampleTime p(0.0000) measurements" https://benchmarksgame-team.pages.debian.net/benchmarksgame/sometimes-people-just-make-up-stuff.html#jvm-startup-time https://benchmarksgame-team.pages.debian.net/benchmarksgame/...