Often, you might want to run servers. If they are JIT compiled, then during warm-up they will be slow(er), and later they will be fast(er). But usually you do

Often, you might want to run servers. If they are JIT compiled, then during warm-up they will be slow(er), and later they will be fast(er). But usually you don't restart servers that often, so they will spend most of their time in "fast mode". It makes sense to me to measure what will be used in practice. Of course, for command line tools such as grep, it is more realistic to measure the warmup phase as well, as that will reflect real usage.