3 ms·

In most cases even beside this issue, microbenchmarks are in general very difficult to interpret for a slew of reasons. I get the impetus to try and isolate a s

In most cases even beside this issue, microbenchmarks are in general very difficult to interpret for a slew of reasons. I get the impetus to try and isolate a single variable, but in practice you're often isolating away real world applicability of the result as well. This is especially true for CPU and I/O benchmarks that typically have an onion's worth of caching layers in hardware and OS.

throwaway2037 3y ago
What you say may be true, but both Oracle and Google have significant investments in parallel microbenchmark tools: Java Microbenchmark Harness and Caliper. Aleksey Shipilëv (@shipilev) famously worked on the Oracle one for a while. He used it extensively in his analysis of core Java code changes. And, I recall that the Google Guava team used Caliper similarly.
- jeffbee 3y ago
  Google has published not just one but two different papers about how and why to ignore microbenchmark signals when making global optimizations. https://people.ucsc.edu/~hlitz/papers/asmdb.pdf https://people.ucsc.edu/~hlitz/papers/asmdb.pdf https://www.usenix.org/system/files/osdi21-hunter.pdf https://www.usenix.org/system/files/osdi21-hunter.pdf
- tveita 3y ago
  I didn't think Caliper was still maintained - looks like there's been some recent commits, but their last release is 2 years ago and says "Caliper is currently in maintenance mode and we don't expect to make changes to it other than bug fixes." https://github.com/google/caliper/releases https://github.com/google/caliper/releases
- marginalia_nu 3y ago
  Mr Shipilev is pretty well aware of this. He used to give a talk "JMH: The Lesser of Two Evils" that goes into some detail about a lot of the problems and pitfalls with microbenchmarks. The argument isn't that microbenchmarks are never useful, but rather that they are usually not very applicable and extremely hard to get right, and even harder to interpret. If you're doing JVM optimizations, it may be a useful tool, but like in practice, it's questionable whether these 12x optimizations will result in even a 0.1% improvement in performance in the median Java application. It's pretty simple when you think about it. If you're optimizing effects that are hard to isolate and in many ways barely measurable in the noise of process memory alignment and similar factors, then in most cases, whatever optimization you make is likewise going to be barely measurable in the noisy hardware/software landscape.
  - re-thc 3y ago
    > it's questionable whether these 12x optimizations will result in even a 0.1% improvement in performance in the median Java application It's good for benchmarks. This is sponsored by Intel so that benchmarks will show x86_64 running faster than arm64 and you should not switch.
    - adgjlsfhk1 3y ago
      except that zen4 has avx and Intel's new consumer chips don't
      - unnah 3y ago
        You're thinking of AVX-512. Current Intel and AMD consumer chips support both AVX and AVX2.
      - kllrnohj 3y ago
        Most zen4 consumer reviews didn't even have any Java benchmarks at all much less any that would have significantly changed the narrative around those CPUs. Epyc vs. Xeon is what this is about as the datacenter is the last holdout for OpenJDK at this point.

throwaway2037 3y ago

What you say may be true, but both Oracle and Google have significant investments in parallel microbenchmark tools: Java Microbenchmark Harness and Caliper. Aleksey Shipilëv (@shipilev) famously worked on the Oracle one for a while. He used it extensively in his analysis of core Java code changes. And, I recall that the Google Guava team used Caliper similarly.

jeffbee 3y ago
Google has published not just one but two different papers about how and why to ignore microbenchmark signals when making global optimizations. https://people.ucsc.edu/~hlitz/papers/asmdb.pdf https://people.ucsc.edu/~hlitz/papers/asmdb.pdf https://www.usenix.org/system/files/osdi21-hunter.pdf https://www.usenix.org/system/files/osdi21-hunter.pdf
tveita 3y ago
I didn't think Caliper was still maintained - looks like there's been some recent commits, but their last release is 2 years ago and says "Caliper is currently in maintenance mode and we don't expect to make changes to it other than bug fixes." https://github.com/google/caliper/releases https://github.com/google/caliper/releases
marginalia_nu 3y ago
Mr Shipilev is pretty well aware of this. He used to give a talk "JMH: The Lesser of Two Evils" that goes into some detail about a lot of the problems and pitfalls with microbenchmarks. The argument isn't that microbenchmarks are never useful, but rather that they are usually not very applicable and extremely hard to get right, and even harder to interpret. If you're doing JVM optimizations, it may be a useful tool, but like in practice, it's questionable whether these 12x optimizations will result in even a 0.1% improvement in performance in the median Java application. It's pretty simple when you think about it. If you're optimizing effects that are hard to isolate and in many ways barely measurable in the noise of process memory alignment and similar factors, then in most cases, whatever optimization you make is likewise going to be barely measurable in the noisy hardware/software landscape.
- re-thc 3y ago
  > it's questionable whether these 12x optimizations will result in even a 0.1% improvement in performance in the median Java application It's good for benchmarks. This is sponsored by Intel so that benchmarks will show x86_64 running faster than arm64 and you should not switch.
  - adgjlsfhk1 3y ago
    except that zen4 has avx and Intel's new consumer chips don't
    - unnah 3y ago
      You're thinking of AVX-512. Current Intel and AMD consumer chips support both AVX and AVX2.
    - kllrnohj 3y ago
      Most zen4 consumer reviews didn't even have any Java benchmarks at all much less any that would have significantly changed the narrative around those CPUs. Epyc vs. Xeon is what this is about as the datacenter is the last holdout for OpenJDK at this point.