4 ms·

Does AVX512 still have the issue that it works for microbenchmarks but may pessimize real workloads because it reduces clock speeds?

by rwmj 3y ago

touisteur 3y ago
If you want to deep dive there Travis Downs has written an amazing thing about it https://travisdowns.github.io/blog/2020/08/19/icl-avx512-freq.html https://travisdowns.github.io/blog/2020/08/19/icl-avx512-fre... and you should follow the links down to his github avx-turbo project that actually runs all sorts of 'frequency reducing' benchmarks and it's just clearer like this. For example I noticed that on some Intel Gold 6230 processors (2000+ USD each at the time), not only avx512 did bring down frequency, but also 'dense avx2' (sequences of heavy FMA). Explained a lot about measured latencies for some real-time processes. Can't thank Travis enough for cutting through all the vagueness there.
- _a_a_a_ 3y ago
  I believe the guy occasionally posts here, or has done, under the name BeeOnRope, also on StackOverflow. I printed out some of his posts here about memory bandwidth. The guy really, really knows his stuff.
  - touisteur 3y ago
    I'm still salty that we needed him to put down time and energy in this, when Intel should have done the job... This license stuff was a major engineering tradeoff, a pain in the ass for many people, that should have been explained as clearly as Travis did, from the get go.
dragontamer 3y ago
That was only on Skylake-X, which I admit was like 4 generations of Intel boxes. But its no longer true today. AMD's Zen4 doesn't downclock at all with AVX512 instructions, and modern Intel chips have only like 100Mhz (aka 0.1Ghz) of downclock measured in practice. In any case, AVX512 is certainly a win from a power-performance perspective. You get more compute-work done with far less decoding/other CPU-related internal core stuff. Even on Skylake-X, I'd expect that AVX512 sorting routines will heat up the CPU less than a non-AVX512 sorting routine... and otherwise be superior from a head and/or power limited perspective.
- comex 3y ago
  Sounds like you’d still want a non-AVX512 path for sufficiently small arrays, though. As an extreme case, sorting a 4-element array presumably doesn’t get enough speedup from AVX512 to be worth downclocking by 100Mhz. I wonder where the cutoff is.
  - dragontamer 3y ago
    The 100MHz downclocking is only seen in Icelake, and not in Rocketlake (no perceptive downclocking behavior from any AVX512). So its a lot of effort to think about a problem that's only identified on a relatively small number of server processors, and (probably) won't be a problem moving forward.
pizza234 3y ago
I don't know the details, but for example, "x265 disables AVX-512 support by default due to clock speed regressions on Intel Skylake-X systems"[¹], which means that yes, this is an existing problem. ¹: https://chipsandcheese.com/2023/04/16/codecs-for-the-4k-era-hevc-av1-vvc-and-beyond https://chipsandcheese.com/2023/04/16/codecs-for-the-4k-era-...
- janwas 3y ago
  That's surprising, I would expect that codecs are doing enough work that the faster processing from AVX-512 more than pays for the downclocking. In JPEG XL we saw 1.4x end to end speedup (including this downclocking) and that was for images, which are often less data than videos. A policy of AVX-512-off-by-default seems pessimistic now that Intel Icelake+ have basically no throttling, and AMD Zen4 definitely none.
  - touisteur 3y ago
    The thing is now your codec (or libc as it was once!) is having a complex side effect. The clamping and ramping back up of the frequency is not immediate and it has an impact of overall latency on the core it's running. I even saw once funny traces on a heavily loaded machine where the avx512 code was rescheduled on other cores almost all the time, because overall the core running the avx512 code seemed to 'be more loaded' and the scheduler would find a new home for the avx512-crunching process. Kind of a 'fun' runaway process.
    - janwas 3y ago
      Fair point :) I wonder why the kernel is so aggressive about migrating threads, it also loses the cache contents. FWIW our command line tools at one point pinned themselves to the current core, which did help performance.
      - gpderetta 3y ago
        OS kernels should be relatively sticky. Possibly it was power/temperature management where the load was moved to an unloaded core to even out power dissipation?
saagarjha 3y ago
Not for several years. If you’re still rocking Skylake you may have issues but beyond that, no, not really.
marginalia_nu 3y ago
In most cases even beside this issue, microbenchmarks are in general very difficult to interpret for a slew of reasons. I get the impetus to try and isolate a single variable, but in practice you're often isolating away real world applicability of the result as well. This is especially true for CPU and I/O benchmarks that typically have an onion's worth of caching layers in hardware and OS.
- throwaway2037 3y ago
  What you say may be true, but both Oracle and Google have significant investments in parallel microbenchmark tools: Java Microbenchmark Harness and Caliper. Aleksey Shipilëv (@shipilev) famously worked on the Oracle one for a while. He used it extensively in his analysis of core Java code changes. And, I recall that the Google Guava team used Caliper similarly.
  - jeffbee 3y ago
    Google has published not just one but two different papers about how and why to ignore microbenchmark signals when making global optimizations. https://people.ucsc.edu/~hlitz/papers/asmdb.pdf https://people.ucsc.edu/~hlitz/papers/asmdb.pdf https://www.usenix.org/system/files/osdi21-hunter.pdf https://www.usenix.org/system/files/osdi21-hunter.pdf
  - tveita 3y ago
    I didn't think Caliper was still maintained - looks like there's been some recent commits, but their last release is 2 years ago and says "Caliper is currently in maintenance mode and we don't expect to make changes to it other than bug fixes." https://github.com/google/caliper/releases https://github.com/google/caliper/releases
  - marginalia_nu 3y ago
    Mr Shipilev is pretty well aware of this. He used to give a talk "JMH: The Lesser of Two Evils" that goes into some detail about a lot of the problems and pitfalls with microbenchmarks. The argument isn't that microbenchmarks are never useful, but rather that they are usually not very applicable and extremely hard to get right, and even harder to interpret. If you're doing JVM optimizations, it may be a useful tool, but like in practice, it's questionable whether these 12x optimizations will result in even a 0.1% improvement in performance in the median Java application. It's pretty simple when you think about it. If you're optimizing effects that are hard to isolate and in many ways barely measurable in the noise of process memory alignment and similar factors, then in most cases, whatever optimization you make is likewise going to be barely measurable in the noisy hardware/software landscape.

Does AVX512 still have the issue that it works for microbenchmarks but may pessimize real workloads because it reduces clock speeds?

by rwmj 3y ago

Does AVX512 still have the issue that it works for microbenchmarks but may pessimize real workloads because it reduces clock speeds?

touisteur 3y ago

If you want to deep dive there Travis Downs has written an amazing thing about it https://travisdowns.github.io/blog/2020/08/19/icl-avx512-freq.html https://travisdowns.github.io/blog/2020/08/19/icl-avx512-fre... and you should follow the links down to his github avx-turbo project that actually runs all sorts of 'frequency reducing' benchmarks and it's just clearer like this. For example I noticed that on some Intel Gold 6230 processors (2000+ USD each at the time), not only avx512 did bring down frequency, but also 'dense avx2' (sequences of heavy FMA). Explained a lot about measured latencies for some real-time processes. Can't thank Travis enough for cutting through all the vagueness there.

_a_a_a_ 3y ago
I believe the guy occasionally posts here, or has done, under the name BeeOnRope, also on StackOverflow. I printed out some of his posts here about memory bandwidth. The guy really, really knows his stuff.
- touisteur 3y ago
  I'm still salty that we needed him to put down time and energy in this, when Intel should have done the job... This license stuff was a major engineering tradeoff, a pain in the ass for many people, that should have been explained as clearly as Travis did, from the get go.

dragontamer 3y ago

That was only on Skylake-X, which I admit was like 4 generations of Intel boxes. But its no longer true today. AMD's Zen4 doesn't downclock at all with AVX512 instructions, and modern Intel chips have only like 100Mhz (aka 0.1Ghz) of downclock measured in practice. In any case, AVX512 is certainly a win from a power-performance perspective. You get more compute-work done with far less decoding/other CPU-related internal core stuff. Even on Skylake-X, I'd expect that AVX512 sorting routines will heat up the CPU less than a non-AVX512 sorting routine... and otherwise be superior from a head and/or power limited perspective.

comex 3y ago
Sounds like you’d still want a non-AVX512 path for sufficiently small arrays, though. As an extreme case, sorting a 4-element array presumably doesn’t get enough speedup from AVX512 to be worth downclocking by 100Mhz. I wonder where the cutoff is.
- dragontamer 3y ago
  The 100MHz downclocking is only seen in Icelake, and not in Rocketlake (no perceptive downclocking behavior from any AVX512). So its a lot of effort to think about a problem that's only identified on a relatively small number of server processors, and (probably) won't be a problem moving forward.

pizza234 3y ago

I don't know the details, but for example, "x265 disables AVX-512 support by default due to clock speed regressions on Intel Skylake-X systems"[¹], which means that yes, this is an existing problem. ¹: https://chipsandcheese.com/2023/04/16/codecs-for-the-4k-era-hevc-av1-vvc-and-beyond https://chipsandcheese.com/2023/04/16/codecs-for-the-4k-era-...

janwas 3y ago
That's surprising, I would expect that codecs are doing enough work that the faster processing from AVX-512 more than pays for the downclocking. In JPEG XL we saw 1.4x end to end speedup (including this downclocking) and that was for images, which are often less data than videos. A policy of AVX-512-off-by-default seems pessimistic now that Intel Icelake+ have basically no throttling, and AMD Zen4 definitely none.
- touisteur 3y ago
  The thing is now your codec (or libc as it was once!) is having a complex side effect. The clamping and ramping back up of the frequency is not immediate and it has an impact of overall latency on the core it's running. I even saw once funny traces on a heavily loaded machine where the avx512 code was rescheduled on other cores almost all the time, because overall the core running the avx512 code seemed to 'be more loaded' and the scheduler would find a new home for the avx512-crunching process. Kind of a 'fun' runaway process.
  - janwas 3y ago
    Fair point :) I wonder why the kernel is so aggressive about migrating threads, it also loses the cache contents. FWIW our command line tools at one point pinned themselves to the current core, which did help performance.
    - gpderetta 3y ago
      OS kernels should be relatively sticky. Possibly it was power/temperature management where the load was moved to an unloaded core to even out power dissipation?

saagarjha 3y ago

Not for several years. If you’re still rocking Skylake you may have issues but beyond that, no, not really.

marginalia_nu 3y ago

In most cases even beside this issue, microbenchmarks are in general very difficult to interpret for a slew of reasons. I get the impetus to try and isolate a single variable, but in practice you're often isolating away real world applicability of the result as well. This is especially true for CPU and I/O benchmarks that typically have an onion's worth of caching layers in hardware and OS.

throwaway2037 3y ago
What you say may be true, but both Oracle and Google have significant investments in parallel microbenchmark tools: Java Microbenchmark Harness and Caliper. Aleksey Shipilëv (@shipilev) famously worked on the Oracle one for a while. He used it extensively in his analysis of core Java code changes. And, I recall that the Google Guava team used Caliper similarly.
- jeffbee 3y ago
  Google has published not just one but two different papers about how and why to ignore microbenchmark signals when making global optimizations. https://people.ucsc.edu/~hlitz/papers/asmdb.pdf https://people.ucsc.edu/~hlitz/papers/asmdb.pdf https://www.usenix.org/system/files/osdi21-hunter.pdf https://www.usenix.org/system/files/osdi21-hunter.pdf
- tveita 3y ago
  I didn't think Caliper was still maintained - looks like there's been some recent commits, but their last release is 2 years ago and says "Caliper is currently in maintenance mode and we don't expect to make changes to it other than bug fixes." https://github.com/google/caliper/releases https://github.com/google/caliper/releases
- marginalia_nu 3y ago
  Mr Shipilev is pretty well aware of this. He used to give a talk "JMH: The Lesser of Two Evils" that goes into some detail about a lot of the problems and pitfalls with microbenchmarks. The argument isn't that microbenchmarks are never useful, but rather that they are usually not very applicable and extremely hard to get right, and even harder to interpret. If you're doing JVM optimizations, it may be a useful tool, but like in practice, it's questionable whether these 12x optimizations will result in even a 0.1% improvement in performance in the median Java application. It's pretty simple when you think about it. If you're optimizing effects that are hard to isolate and in many ways barely measurable in the noise of process memory alignment and similar factors, then in most cases, whatever optimization you make is likewise going to be barely measurable in the noisy hardware/software landscape.