4 ms·

> Clojure, incidentally has great benchmarking due to being on the JVM. Eh? Benchmarking on the JVM is notoriously difficult bordering on impossible. There's t

> Clojure, incidentally has great benchmarking due to being on the JVM. Eh? Benchmarking on the JVM is notoriously difficult bordering on impossible. There's things like Google Caliper but test runs take forever due to attempting to force JIT warmup and doing GCs after every run. And the project's own wiki tells you the results are basically meaningless for a variety of reasons. Benchmarking things like C++ or Rust are trivial by comparison since when you call method foo() it's gonna do pretty much the same instructions every time. Highly consistent, highly repeatable, highly benchmarkable. Call method foo() in a JVM language and there's not a single person on the planet that can reliably tell you what's going to get executed on the metal. That's why you typically profile JVM languages rather than benchmarking them.

mcguire 9y ago
Benchmarking is pretty meaningfree in the best cases. The JVM isn't any worse in this regard. The JVM is slow in many regimes due to things like JIT warmup. Rather than accepting that many JVM people are hypersensitive: thou must only benchmark in the JVM's best case scenario.
- kllrnohj 9y ago
  > Benchmarking is pretty meaningfree in the best cases. That's completely false. Benchmarking is a core staple of building & evaluating performance-sensitive libraries or other routines. It doesn't work (well) in non-deterministic languages which reduces its usefulness scope, but in things like C++ it's highly useful and reliable for evaluation of libraries and monitoring for regressions. > The JVM is slow in many regimes due to things like JIT warmup. Rather than accepting that many JVM people are hypersensitive: thou must only benchmark in the JVM's best case scenario. Well it's not just the JIT that's a problem. It's also things like GC passes. Does that get included in the results or not? Do you force GC passes between runs? How about finalizers? The answers to those questions depends on the state of the rest of the system and the expected workload, it's not something you can just trivially answer or even accommodate in a framework since most of the behavior is up to the particular implementation, which can then further vary based off of command line flags.
modalduality 9y ago
My bad, I meant profiling, not benchmarking.

mcguire 9y ago

Benchmarking is pretty meaningfree in the best cases. The JVM isn't any worse in this regard. The JVM is slow in many regimes due to things like JIT warmup. Rather than accepting that many JVM people are hypersensitive: thou must only benchmark in the JVM's best case scenario.

kllrnohj 9y ago
> Benchmarking is pretty meaningfree in the best cases. That's completely false. Benchmarking is a core staple of building & evaluating performance-sensitive libraries or other routines. It doesn't work (well) in non-deterministic languages which reduces its usefulness scope, but in things like C++ it's highly useful and reliable for evaluation of libraries and monitoring for regressions. > The JVM is slow in many regimes due to things like JIT warmup. Rather than accepting that many JVM people are hypersensitive: thou must only benchmark in the JVM's best case scenario. Well it's not just the JIT that's a problem. It's also things like GC passes. Does that get included in the results or not? Do you force GC passes between runs? How about finalizers? The answers to those questions depends on the state of the rest of the system and the expected workload, it's not something you can just trivially answer or even accommodate in a framework since most of the behavior is up to the particular implementation, which can then further vary based off of command line flags.

modalduality 9y ago

My bad, I meant profiling, not benchmarking.