7 ms·

AVX-512: First Impressions on Performance and Programmability

125 points by shihab 9mo ago

fithisux 9mo ago
What I get in these article is that the original intent on C language stands true. Use C as a common platform denominator without crazy optimizations (like tcc). If you need performance, specialize, C gives you the tools to call assembly (or use compiler some intrinsic or even inline assembly). Complex compiler doing crazy optimizations, in my opinion, is not worth it.
- eru 9mo ago
  Well, C is a lie anyway: it's not how computers work any more (and I'm not sure it's how they ever worked).
  - woooooo 9mo ago
    Assembly isnt how they work under the hood, but its the highest fidelity interface we have. C as "portable assembler" is targeting the surface that chip designers give us and the one that they try to make fast via all their microcode tricks.
- kergonath 9mo ago
  > Complex compiler doing crazy optimizations, in my opinion, is not worth it. For these optimisations that are in the back-end, they are used for other languages that can be higher-level or that cannot drop to assembler as easily. C is just one of the front-ends of modern compiler suites.
  - ziml77 9mo ago
    Some of them even give you access to this stuff. C# has Vector<T> which picks implementations of various operations based on the current CPU's capabilities. But they also let you have complete control if you want by providing functions in the System.Runtime.Instrinsics namespace.
- vjerancrnjak 9mo ago
  I remember being quite surprised that my implementation which uses manual stack updates is much slower than what compiler had with recursion. Turns out, I was pushing and popping from stack on every conceptual "recursive call", but compiler figured out it can keep 2-3 recursive levels in registers and pop/push 30% of the time, had more stuff in memory than my version as well. Even when I reduced memory read/writes to ~50% of the recursive program, kept most of the state in registers, the recursive program was faster anyway due to just using more registers than me. I realized then that I cannot reason about the microoptimizations at all if I'm coding in a high-level language like C or C++. Hard to predict the CPU pipeline, sometimes profile guided optimization gets me there faster than my own silliness of assuming I can reason about it.
pjmlp 9mo ago
> In CPU world there is a desire to shield programmers from those low-level details, but I think there are two interesting forces at play now-a-days that’ll change it soon. On one hand, Dennard Scaling (aka free lunch) is long gone, hardware landscape is getting increasingly fragmented and specialized out of necessity, software abstractions are getting leakier, forcing developers to be aware of the lowest levels of abstraction, hardware, for good performance. The problem is that not all programming languages expose SIMD, and even if they do it is only a portable subset, additionally the kind of skills that are required to be able to use SIMD properly isn't something everyone is confortable doing. I certainly am not, still managed to get around with MMX and early SSE, can manage shading languages, and that is about it.
- adgjlsfhk1 9mo ago
  The good news is that the portable subset of SIMD is all you really need anyway. If you go beyond the portable subset, you need per-architecture code writing and testing, and you're mostly talking about pretty small gains relative to the cost.
camel-cdr 9mo ago
> The answer, if it’s not obvious from my tone already:), is 8%. Not if the data is small and in cache. > The performant route with AVX-512 would probably include the instruction vpconflictd, but I couldn’t really find any elegant way to use it. I think the best way to do this is duplicate sum_r and count 16 times, so each pane has a seperate accumulation bucket and there can't be any conflicts. After the loop, you quickly do a sum reduction for each of the 16 buckets.
- shihab 9mo ago
  Yeah N is big enough that entire data isn't in the cache, but the memory access pattern here is the next best thing: totally linear, predictable access. I remember seeing around 94%+ L1d cache hit rate.
- praptak 9mo ago
  > Not if the data is small and in cache. Isn't it another way of saying what the author says in the previous paragraph, namely that "ideal SIMD speedup can only come from problems that are compute bound"? If the cost of getting the input data into the cache is already large compared to processing it with the non-vectorized code, then SIMD cannot achieve meaningful speedup. The opposite of this condition (processing is expensive compared to the cost of data into the cache) is basically the definition "compute bound".
chillitom 9mo ago
Initial example takes array pointers without the __restrict__ keyword/extension so compiler might assume they could be aliased to same address space and will code defensively. Would be interesting to see if auto vec performs better with that addition.
- chillitom 9mo ago
  Also trying to let the compilers know that the float* are aligned would be a good move. auto aligned_p = std::assume_aligned<16>(p)
  - Remnant44 9mo ago
    which honestly, shouldn't be neccessary today with avx512. There's essentially no reason to prefer the aligned load/store commands over the unaligned ones - if the actual pointer is unaligned it will function correctly at half the throughput, while if it_is_ aligned you will get the same performance as the aligned-only load. No reason for the compiler to balk at vectorizing unaligned data these days.
    - dmpk2k 9mo ago
      > There's essentially no reason to prefer the aligned load/store commands over the unaligned ones - if the actual pointer is unaligned it will function correctly at half the throughput Getting a fault instead of half the performance is actually a really good reason to prefer aligned load/store. To be fair, you're talking about a compiler here, but I never understood why people use the unaligned intrinsics...
      - Remnant44 9mo ago
        There are many situations where your data is essentially _majority_ unaligned. Considerable effort by the hardware guys has gone into making that situation work well. A great example would be a convolution-kernel style code - with AVX512 you are using 64 bytes at a time (a whole cacheline), and sampling a +- N element neighborhood around a pixel. By definition most of those reads will be unaligned! A lot of other great use cases for SIMD don't let you dictate the buffer alignment. If the code is constrained by bandwidth over compute, I have found it to be worth doing a head/body/tail situation where you do one misaligned iteration before doing the bulk of the work in alignment, but honestly for that to be worth it you have to be working almost completely out of L1 cache which is rare... otherwise you're going to be slowed down to L2 or memory speed anyways, at which point the half rate penalty doesn't really matter. The early SSE-style instructions often favored making two aligned reads and then extracting your sliding window from that, but there's just no point doing that on modern hardware - it will be slower.
ecesena 9mo ago
If you have the opportunity, try out a zen5. Significant improvements. See also https://www.numberworld.org/blogs/2024_8_7_zen5_avx512_teardown/ https://www.numberworld.org/blogs/2024_8_7_zen5_avx512_teard...
nnevatie 9mo ago
I found this a weird article. If you wish to see some speedups using AVX512, without limiting yourself to C or C++, you might want to try ISPC (https://ispc.github.io/index.html https://ispc.github.io/index.html). You'll get sane aliasing rules from the perspective of performance, multi-target binaries with dynamic dispatching and a lot more control over the code generated.
- majke 9mo ago
  Ispc looks interesting. Does it work with amd? They hint on gpu’s , i guess mostly intel ones?
  - dataking 9mo ago
    Yes, it works with AMD CPUs as well as various ARM ones, e.g. Apple silicon. See for instance https://github.com/ispc/ispc/pull/2160 https://github.com/ispc/ispc/pull/2160
  - nnevatie 9mo ago
    Yes, works well with AMD. You can compile multi-target so that you'll have e.g. SSE4.2, AVX2, AVX512 support built to your binaries and the best (widest) version is picked by the runtime automatically.
- theowaway 9mo ago
  ispc is something that deserves to be much more widely known about- it does an excellent job of bringing the cuda programming model to cpus
  - grumbelbart2 9mo ago
    Is there a way to compile it to something else than x86, like arm/aarch64?
    - nnevatie 9mo ago
      > It currently supports multiple flavours of x86 (SSE2, SSE4, AVX, AVX2, and AVX512), ARM (NEON), and Intel® GPU architectures (Xe family).
DeathArrow 9mo ago
>On one hand, Dennard Scaling (aka free lunch) is long gone, hardware landscape is getting increasingly fragmented and specialized out of necessity, software abstractions are getting leakier, forcing developers to be aware of the lowest levels of abstraction, hardware, for good performance. There are lots of people using Javascript frameworks to build slow desktop and mobile software.
- user_7832 9mo ago
  I wonder if the excess CO2 emitted by devices around the world using bloated software that has no need to be so (hullo MS Teams) could be calculated in terms of # of cross atlantic voyages of jets.
physicsguy 9mo ago
A few gentle points: (a) You mention that the NVidia docs push people to use libraries, etc. to really get high performance CUDA kernels, rather than writing them themselves. My argument would be that SIMD is exactly the same - they're something really that are perfect if you're writing a BLAS implementation but are too low level for most developers thinking about a problem to make use of. (b) You show a problem where autovectorisation fails because of branching, and jump straight to intrinsics as the solution which you basically say are ugly. Looking at the intrinsic code, you're using a mask to deal with the branching. But there's a middle ground - almost always you would want to try restructuring the problem, e.g. splitting up loops and adding masks where there's conditions - i.e. lean into the SIMD paradigm. This would also be the same advice in CUDA. (c) As you've found, GCC actually performs quite poorly for x86-64 optimisations compared to Intel. It's not always clear cut though, the Intel Compiler for e.g. sacrifices IEEE 764 float precision and go down to ~14 digits of precision in it's defaults, because it sets the flag `-fp-model=fast -fma`. This is true of both the legacy and new Intel compiler. If you switch to `-fp-model=strict` then you may find that the results are closer. (d) AVX512 is quite hardware specific. Some processors execute these instructions much better than others. It's really a collection of extensions, and you get frequency downclocking that's better/worse on different processors as these instructions are executed.
- mort96 9mo ago
  Regarding (b), I would never rely on auto vectorization because I have no insight into it. The only way to check if my code is written such that auto-vectorization can do its thing is to compile my program with every combination of compiler, optimization setting and platform I intend to support, disassemble all the resulting binaries, and analyze the disassembly to try to figure out if it did autovectorization in the way I expect. That's a terrible developer experience; writing intrinsics by hand is much easier, and more robust. I'd need to re-check every piece of autovectorized code after every change and after every compiler upgrade. I just treat autovectorization like I treat every other fancy optimization that's not just constant folding and inlining: nice when it happens to work out, it probably happens to make my code a couple percent faster on average, but I absolutely can't rely on it in any place where I depend on the performance.
saagarjha 9mo ago
Isn’t k-means memory bandwidth bound? What was the arithmetic intensity of the final code?
- NohatCoder 9mo ago
  I assume that the image would at least fit in L3.
- shihab 9mo ago
  No. Assuming `k` is small enough, which in practice often is, the arithmetic intensity of this kernel is 25-90 Flops/Byte, way above the roofline knee of any modern CPU.
BoredomIsFun 9mo ago
A factoid - earlier batches of Alder Lake 12th gen consumer CPU has a rare AVX512 _FP16_ extension. Afaik it was very very fast.
alecco 9mo ago
> CUDA architects [...] happily exposed every ugly details of underlying hardware to the programmer if that allows a bit more performance. After spending more than a decade dancing around all the underlying x86 hidden stuff for low-level optimization, I appreciate CUDA a lot. Everything is there under your total control. No more one size fits all. Higher barrier of entry but no surprises and less time spent debugging to figure out what landmine your code stepped into.
- user3939382 9mo ago
  Then you’re locked into the ecosystem and whims of signed proprietary drivers so in a way you have no control whatsoever.
  - alecco 9mo ago
    Sure. But Intel's beancounter board is way more scary [1] and moving from CUDA to AMD's ROCm isn't that hard, anyway. [1] "Intel Officially Introduces Pay-As-You-Go Chip Licensing - Intel's Xeon Sapphire Rapids CPUs to activate additional features on demand" https://www.tomshardware.com/news/intel-officially-introduces-pay-as-you-go-chip-licensing https://www.tomshardware.com/news/intel-officially-introduce... (2022)
randomint64 9mo ago
For those who want to get started with SIMD programming in Rust, here is a great resource: https://kerkour.com/introduction-rust-simd https://kerkour.com/introduction-rust-simd

125 points by shihab 9mo ago

What I get in these article is that the original intent on C language stands true. Use C as a common platform denominator without crazy optimizations (like tcc). If you need performance, specialize, C gives you the tools to call assembly (or use compiler some intrinsic or even inline assembly). Complex compiler doing crazy optimizations, in my opinion, is not worth it.

eru 9mo ago
Well, C is a lie anyway: it's not how computers work any more (and I'm not sure it's how they ever worked).
- woooooo 9mo ago
  Assembly isnt how they work under the hood, but its the highest fidelity interface we have. C as "portable assembler" is targeting the surface that chip designers give us and the one that they try to make fast via all their microcode tricks.
kergonath 9mo ago
> Complex compiler doing crazy optimizations, in my opinion, is not worth it. For these optimisations that are in the back-end, they are used for other languages that can be higher-level or that cannot drop to assembler as easily. C is just one of the front-ends of modern compiler suites.
- ziml77 9mo ago
  Some of them even give you access to this stuff. C# has Vector<T> which picks implementations of various operations based on the current CPU's capabilities. But they also let you have complete control if you want by providing functions in the System.Runtime.Instrinsics namespace.
vjerancrnjak 9mo ago
I remember being quite surprised that my implementation which uses manual stack updates is much slower than what compiler had with recursion. Turns out, I was pushing and popping from stack on every conceptual "recursive call", but compiler figured out it can keep 2-3 recursive levels in registers and pop/push 30% of the time, had more stuff in memory than my version as well. Even when I reduced memory read/writes to ~50% of the recursive program, kept most of the state in registers, the recursive program was faster anyway due to just using more registers than me. I realized then that I cannot reason about the microoptimizations at all if I'm coding in a high-level language like C or C++. Hard to predict the CPU pipeline, sometimes profile guided optimization gets me there faster than my own silliness of assuming I can reason about it.

pjmlp 9mo ago

> In CPU world there is a desire to shield programmers from those low-level details, but I think there are two interesting forces at play now-a-days that’ll change it soon. On one hand, Dennard Scaling (aka free lunch) is long gone, hardware landscape is getting increasingly fragmented and specialized out of necessity, software abstractions are getting leakier, forcing developers to be aware of the lowest levels of abstraction, hardware, for good performance. The problem is that not all programming languages expose SIMD, and even if they do it is only a portable subset, additionally the kind of skills that are required to be able to use SIMD properly isn't something everyone is confortable doing. I certainly am not, still managed to get around with MMX and early SSE, can manage shading languages, and that is about it.

adgjlsfhk1 9mo ago
The good news is that the portable subset of SIMD is all you really need anyway. If you go beyond the portable subset, you need per-architecture code writing and testing, and you're mostly talking about pretty small gains relative to the cost.

camel-cdr 9mo ago

> The answer, if it’s not obvious from my tone already:), is 8%. Not if the data is small and in cache. > The performant route with AVX-512 would probably include the instruction vpconflictd, but I couldn’t really find any elegant way to use it. I think the best way to do this is duplicate sum_r and count 16 times, so each pane has a seperate accumulation bucket and there can't be any conflicts. After the loop, you quickly do a sum reduction for each of the 16 buckets.

shihab 9mo ago
Yeah N is big enough that entire data isn't in the cache, but the memory access pattern here is the next best thing: totally linear, predictable access. I remember seeing around 94%+ L1d cache hit rate.
praptak 9mo ago
> Not if the data is small and in cache. Isn't it another way of saying what the author says in the previous paragraph, namely that "ideal SIMD speedup can only come from problems that are compute bound"? If the cost of getting the input data into the cache is already large compared to processing it with the non-vectorized code, then SIMD cannot achieve meaningful speedup. The opposite of this condition (processing is expensive compared to the cost of data into the cache) is basically the definition "compute bound".

chillitom 9mo ago

Initial example takes array pointers without the __restrict__ keyword/extension so compiler might assume they could be aliased to same address space and will code defensively. Would be interesting to see if auto vec performs better with that addition.

chillitom 9mo ago
Also trying to let the compilers know that the float* are aligned would be a good move. auto aligned_p = std::assume_aligned<16>(p)
- Remnant44 9mo ago
  which honestly, shouldn't be neccessary today with avx512. There's essentially no reason to prefer the aligned load/store commands over the unaligned ones - if the actual pointer is unaligned it will function correctly at half the throughput, while if it_is_ aligned you will get the same performance as the aligned-only load. No reason for the compiler to balk at vectorizing unaligned data these days.
  - dmpk2k 9mo ago
    > There's essentially no reason to prefer the aligned load/store commands over the unaligned ones - if the actual pointer is unaligned it will function correctly at half the throughput Getting a fault instead of half the performance is actually a really good reason to prefer aligned load/store. To be fair, you're talking about a compiler here, but I never understood why people use the unaligned intrinsics...
    - Remnant44 9mo ago
      There are many situations where your data is essentially _majority_ unaligned. Considerable effort by the hardware guys has gone into making that situation work well. A great example would be a convolution-kernel style code - with AVX512 you are using 64 bytes at a time (a whole cacheline), and sampling a +- N element neighborhood around a pixel. By definition most of those reads will be unaligned! A lot of other great use cases for SIMD don't let you dictate the buffer alignment. If the code is constrained by bandwidth over compute, I have found it to be worth doing a head/body/tail situation where you do one misaligned iteration before doing the bulk of the work in alignment, but honestly for that to be worth it you have to be working almost completely out of L1 cache which is rare... otherwise you're going to be slowed down to L2 or memory speed anyways, at which point the half rate penalty doesn't really matter. The early SSE-style instructions often favored making two aligned reads and then extracting your sliding window from that, but there's just no point doing that on modern hardware - it will be slower.

ecesena 9mo ago

If you have the opportunity, try out a zen5. Significant improvements. See also https://www.numberworld.org/blogs/2024_8_7_zen5_avx512_teardown/ https://www.numberworld.org/blogs/2024_8_7_zen5_avx512_teard...

nnevatie 9mo ago

I found this a weird article. If you wish to see some speedups using AVX512, without limiting yourself to C or C++, you might want to try ISPC (https://ispc.github.io/index.html https://ispc.github.io/index.html). You'll get sane aliasing rules from the perspective of performance, multi-target binaries with dynamic dispatching and a lot more control over the code generated.

majke 9mo ago
Ispc looks interesting. Does it work with amd? They hint on gpu’s , i guess mostly intel ones?
- dataking 9mo ago
  Yes, it works with AMD CPUs as well as various ARM ones, e.g. Apple silicon. See for instance https://github.com/ispc/ispc/pull/2160 https://github.com/ispc/ispc/pull/2160
- nnevatie 9mo ago
  Yes, works well with AMD. You can compile multi-target so that you'll have e.g. SSE4.2, AVX2, AVX512 support built to your binaries and the best (widest) version is picked by the runtime automatically.
theowaway 9mo ago
ispc is something that deserves to be much more widely known about- it does an excellent job of bringing the cuda programming model to cpus
- grumbelbart2 9mo ago
  Is there a way to compile it to something else than x86, like arm/aarch64?
  - nnevatie 9mo ago
    > It currently supports multiple flavours of x86 (SSE2, SSE4, AVX, AVX2, and AVX512), ARM (NEON), and Intel® GPU architectures (Xe family).

DeathArrow 9mo ago

>On one hand, Dennard Scaling (aka free lunch) is long gone, hardware landscape is getting increasingly fragmented and specialized out of necessity, software abstractions are getting leakier, forcing developers to be aware of the lowest levels of abstraction, hardware, for good performance. There are lots of people using Javascript frameworks to build slow desktop and mobile software.

user_7832 9mo ago
I wonder if the excess CO2 emitted by devices around the world using bloated software that has no need to be so (hullo MS Teams) could be calculated in terms of # of cross atlantic voyages of jets.

physicsguy 9mo ago

A few gentle points: (a) You mention that the NVidia docs push people to use libraries, etc. to really get high performance CUDA kernels, rather than writing them themselves. My argument would be that SIMD is exactly the same - they're something really that are perfect if you're writing a BLAS implementation but are too low level for most developers thinking about a problem to make use of. (b) You show a problem where autovectorisation fails because of branching, and jump straight to intrinsics as the solution which you basically say are ugly. Looking at the intrinsic code, you're using a mask to deal with the branching. But there's a middle ground - almost always you would want to try restructuring the problem, e.g. splitting up loops and adding masks where there's conditions - i.e. lean into the SIMD paradigm. This would also be the same advice in CUDA. (c) As you've found, GCC actually performs quite poorly for x86-64 optimisations compared to Intel. It's not always clear cut though, the Intel Compiler for e.g. sacrifices IEEE 764 float precision and go down to ~14 digits of precision in it's defaults, because it sets the flag `-fp-model=fast -fma`. This is true of both the legacy and new Intel compiler. If you switch to `-fp-model=strict` then you may find that the results are closer. (d) AVX512 is quite hardware specific. Some processors execute these instructions much better than others. It's really a collection of extensions, and you get frequency downclocking that's better/worse on different processors as these instructions are executed.

mort96 9mo ago
Regarding (b), I would never rely on auto vectorization because I have no insight into it. The only way to check if my code is written such that auto-vectorization can do its thing is to compile my program with every combination of compiler, optimization setting and platform I intend to support, disassemble all the resulting binaries, and analyze the disassembly to try to figure out if it did autovectorization in the way I expect. That's a terrible developer experience; writing intrinsics by hand is much easier, and more robust. I'd need to re-check every piece of autovectorized code after every change and after every compiler upgrade. I just treat autovectorization like I treat every other fancy optimization that's not just constant folding and inlining: nice when it happens to work out, it probably happens to make my code a couple percent faster on average, but I absolutely can't rely on it in any place where I depend on the performance.

saagarjha 9mo ago

Isn’t k-means memory bandwidth bound? What was the arithmetic intensity of the final code?

NohatCoder 9mo ago
I assume that the image would at least fit in L3.
shihab 9mo ago
No. Assuming `k` is small enough, which in practice often is, the arithmetic intensity of this kernel is 25-90 Flops/Byte, way above the roofline knee of any modern CPU.

BoredomIsFun 9mo ago

A factoid - earlier batches of Alder Lake 12th gen consumer CPU has a rare AVX512 _FP16_ extension. Afaik it was very very fast.

alecco 9mo ago

> CUDA architects [...] happily exposed every ugly details of underlying hardware to the programmer if that allows a bit more performance. After spending more than a decade dancing around all the underlying x86 hidden stuff for low-level optimization, I appreciate CUDA a lot. Everything is there under your total control. No more one size fits all. Higher barrier of entry but no surprises and less time spent debugging to figure out what landmine your code stepped into.

user3939382 9mo ago
Then you’re locked into the ecosystem and whims of signed proprietary drivers so in a way you have no control whatsoever.
- alecco 9mo ago
  Sure. But Intel's beancounter board is way more scary [1] and moving from CUDA to AMD's ROCm isn't that hard, anyway. [1] "Intel Officially Introduces Pay-As-You-Go Chip Licensing - Intel's Xeon Sapphire Rapids CPUs to activate additional features on demand" https://www.tomshardware.com/news/intel-officially-introduces-pay-as-you-go-chip-licensing https://www.tomshardware.com/news/intel-officially-introduce... (2022)

randomint64 9mo ago

For those who want to get started with SIMD programming in Rust, here is a great resource: https://kerkour.com/introduction-rust-simd https://kerkour.com/introduction-rust-simd