3 ms·
One problem is that testing these functions in isolation and with synthetic data is not the same thing as testing them in your real program and with real data.
by chrisseaton 9y ago
One problem is that testing these functions in isolation and with synthetic data is not the same thing as testing them in your real program and with real data. Modern JIT compilers like Graal are extremely sophisticated and will look at your program in ways that are not easy to understand.
Even just preventing a powerful compiler like Graal from optimising away your microbenchmark is going to be a challenge - Graal currently defeats the state-of-the-art JMH for example, and most people wouldn't even going as far as to use JMH.
I have many examples of microbenchmarks that Graal (when being used as a JIT for Ruby in my case) will optimise away that people will find very surprising.
https://twitter.com/chrisgseaton/status/619885182104043520 https://twitter.com/chrisgseaton/status/619885182104043520