6 ms·

> Numbers speak for themselves No, microbenchmarks never speak for themselves.

jcelerier 9y ago
What if the only code you run is microcode ? IF 60-70% of a given run-time ends up in the same few small dozen functions, it's certainly worth benchmarking them.
- chrisseaton 9y ago
  One problem is that testing these functions in isolation and with synthetic data is not the same thing as testing them in your real program and with real data. Modern JIT compilers like Graal are extremely sophisticated and will look at your program in ways that are not easy to understand. Even just preventing a powerful compiler like Graal from optimising away your microbenchmark is going to be a challenge - Graal currently defeats the state-of-the-art JMH for example, and most people wouldn't even going as far as to use JMH. I have many examples of microbenchmarks that Graal (when being used as a JIT for Ruby in my case) will optimise away that people will find very surprising. https://twitter.com/chrisgseaton/status/619885182104043520 https://twitter.com/chrisgseaton/status/619885182104043520
- Veedrac 9y ago
  By no means do I think microbenchmarks convey no useful information. My point is that they are never useful data in isolation, because the only time you can act on them is if you know why they are behaving the way they are. In this case it seems like the only thing being measured was String::compareToIgnoreCase, and the difference had nothing to do with the optimizer. In other cases the cause will be something else. I wrote a blog post in response to someone's attempt to investigate different implementations of a small function through a sequence of microbenchmarks without asking why. I go into detail about a bunch of ways they ended up being wrong because they thought the numbers spoke for themselves. This only gets worse when talking about JITs, since their behaviour is even less local. https://medium.com/@veedrac/learning-the-value-of-good-benchmarking-technique-with-c-magic-squares-b61b3386c97f https://medium.com/@veedrac/learning-the-value-of-good-bench...

What if the only code you run is microcode ? IF 60-70% of a given run-time ends up in the same few small dozen functions, it's certainly worth benchmarking them.

chrisseaton 9y ago
One problem is that testing these functions in isolation and with synthetic data is not the same thing as testing them in your real program and with real data. Modern JIT compilers like Graal are extremely sophisticated and will look at your program in ways that are not easy to understand. Even just preventing a powerful compiler like Graal from optimising away your microbenchmark is going to be a challenge - Graal currently defeats the state-of-the-art JMH for example, and most people wouldn't even going as far as to use JMH. I have many examples of microbenchmarks that Graal (when being used as a JIT for Ruby in my case) will optimise away that people will find very surprising. https://twitter.com/chrisgseaton/status/619885182104043520 https://twitter.com/chrisgseaton/status/619885182104043520
Veedrac 9y ago
By no means do I think microbenchmarks convey no useful information. My point is that they are never useful data in isolation, because the only time you can act on them is if you know why they are behaving the way they are. In this case it seems like the only thing being measured was String::compareToIgnoreCase, and the difference had nothing to do with the optimizer. In other cases the cause will be something else. I wrote a blog post in response to someone's attempt to investigate different implementations of a small function through a sequence of microbenchmarks without asking why. I go into detail about a bunch of ways they ended up being wrong because they thought the numbers spoke for themselves. This only gets worse when talking about JITs, since their behaviour is even less local. https://medium.com/@veedrac/learning-the-value-of-good-benchmarking-technique-with-c-magic-squares-b61b3386c97f https://medium.com/@veedrac/learning-the-value-of-good-bench...