4 ms·

Neat and performant code like the article makes me very curious how the competition will shake out between AMD's AVX512 implementation and Intel's upcoming AVX1

Neat and performant code like the article makes me very curious how the competition will shake out between AMD's AVX512 implementation and Intel's upcoming AVX10. The entire point of AVX10 seems to be to resolve Intel's P vs E core situation, while AMD seems to have taken a better approach of using either full width (Zen5) or double-pumped 256bit (Zen4, Zen5 mobile) as appropriate to the situation, while making the API seamless. The big gains delivered in the article are all on a double-pumped Zen4 core! AVX512 brings a lot to the table so its quite frustrating that Intel market-segmented support for it so heavily as to completely inhibit its adoption in broad-based client code.

Tuna-Fish 2y ago
If Intel actually implements AVX10/256 on every CPU they ship going forwards, it will eventually win simply by availability. The market has repeatedly and thoroughly rejected dispatching to different code paths based on CPU, so the only SIMD implementation that actually matters is the lowest common denominator. And since AVX10.1/256 and AVX512VL have a shared subset, that will be what people will eventually target once enough time has passed and nearly everyone has a CPU that can support it. AVX512 will continue to give AMD some easy wins on the few benchmarking apps that were actually updated to support it, but if Intel sticks with the AVX10 plan I expect that AMD will eventually just use the double-pumped SIMD pipes for everything, just because they are the more efficient way to support AVX10/256 while retaining AVX512 compatibility. Intel did a lot of bad choices in the past decade, but segmenting the market based on instruction set has to be one of the worst. They just chose to kill all the momentum and interest in their newest and best innovations. Hopefully they actually add AVX10/256 support to the whole lineup, because the width is the least interesting part about AVX512, the masked operations especially are a lot more important.
- Remnant44 2y ago
  I agree. I still hesitate to ship code that requires even AVX1, even though it was first introduced in 2011(!). AVX512 really improves the instruction set. Not just from masking but from some really big holes filled in terms of instructions available that AVX2 doesn't have a good solution for. At the same time I also DO have plenty of code that could definitely use the compute throughput improvement of 512 bit vectors. But it's definitely a more niche usage. You have to at least nominally satisfy that you: 1) Benefit from 2x the ALU throughput 2) Live mostly in the cache 3) Are not worth running on the GPU instead.
  - Const-me 2y ago
    It depends on the product and the market. I’ve been shipping software that requires AVX2 and FMA3 because it is a professional CAM/CAE application. Our users typically don’t use sub-$100 Intel processors like Pentium and Celeron. The key is communication, you need to make sure users are aware of the hardware requirements before they buy. Another example, in the server market, you can almost always count on having at least AVX2 support because most servers today are cloud-based. For people running these cloud services, power efficiency is crucial due to the high cost of electricity, so they tend to replace old hardware regularly. On the other hand, for desktop software aimed at a broad audience, there are valid reasons to hesitate before shipping code that requires AVX1.
- janwas 2y ago
  Dispatching is actually heavily used. Image/video codecs, cryptography, and ML libraries routinely use it, because the lowest common denominator is very low indeed.
  - Tuna-Fish 2y ago
    The things you listed are probably less than 1% of all client loads. Video codecs and ML mostly run on accelerators and cryptography is a tiny proportion of all loads.
    - richardwhiuk 2y ago
      Surely h264 encode and decode is substantial, given the large amount of video consumed?
      - Const-me 2y ago
        I don’t believe many people encode or especially decode video with CPU-running codes. Modern GPUs include hardware accelerators for popular video codecs, these are typically way more power efficient than even AVX-512 software implementations.
        account42 2y ago
        > I don’t believe many people encode or especially decode video with CPU-running codes. Believe what you want but as soon as realtime is not a concern and quality matters you'll be using CPU-based encoders unless you have special hardware for your use case. > Modern GPUs include hardware accelerators for popular video codecs ... with shit quality encoders because they are designed for speed first. Decoding is a different matter but even there older hardware can easily end up not supporting codecs (or profiles of them) that you come accross.
    - zekica 2y ago
      Cryptography is every download you ever do in a browser, bitlocker or equivalent encrypted disk access. Add to that gzip or brotli compression (common in HTTP). Hardware decoders for newer video codecs (such as AV1) are not that common (less than 50% of devices) so most youtube watch time on desktops/laptops is software decoding. It's a lot more than 1%.
- School-Cotton 2y ago
  > The market has repeatedly and thoroughly rejected dispatching to different code paths based on CPU What do you mean? At least numpy and pytorch (the only numeric libraries I'm familiar with) both use runtime dispatching.
- nemetroid 2y ago
  > The market has repeatedly and thoroughly rejected dispatching to different code paths based on CPU, so the only SIMD implementation that actually matters is the lowest common denominator. RHEL just moved up to x86_64-v2, equivalent to 2009 level CPUs. And they’re an early mover, Fedora/Ubuntu/Arch have not done the same. Glibc has used CPU dispatching for str* and mem* functions for over a decade.
  - IWeldMelons 2y ago
    FreeBSD added AVX512 dispatch into it C library AFAIK.
- TinkersW 2y ago
  I don't get saying mask operations are more important than width? Mask operations can be trivially emulated with vblend, it is one extra instruction.. Width can't be emulated, you just are stuck running half speed. This take keeps getting repeated, but doesn't appear to be backed up by reality. Intel hasn't even put AVX10 on their upcoming chips(skymont), so it appears to be going nowhere.
  - account42 2y ago
    > Mask operations can be trivially emulated with vblend, it is one extra instruction.. For unaligned loads where you can't guarantee that the entire vector is on a mapped page?
  - fanf2 2y ago
    The important feature of AVX-512 demonstrated in my blog post is masked loads and stores, which can't be emulated with vblend.
Aardwolf 2y ago
I don't understand why intel doesn't make their E-cores use double pumped AVX512 to solve this issue (or just make P-core only CPUs for desktops like it should). They have had years to fix this by now. It's annoying that despite AMD's support, market share makes the adoption of this not happen anyway, and the AVX10 thing will unfortunately allow them to hold back the world even longer. What I like to see (for desktop) is: better cores, more cores, well standardized instruction sets that unlock useful things (wide SIMD, float16, gather/scatter, ...). AMD is doing pretty well at this. What Intel is doing instead: put weak cores alongside decent cores, cripple the decent cores to keep up with the weak cores, release CPUs with the same amount of cores as before for many generations in a row, use the weak cores to make it sound like they have more cores than they have, bring out way too many variants of instructions sets to ever have a useful common set, drop support for their own promising sounding instructions I just really dislike anything Intel has come out with or plans to come out with lately :p My cycle of preference of manufacturer (for desktop computers) has been: 90s: Intel. Early 2000s: AMD (pentium 4 was so meh). late 2000's+2010s: Intel. Now: AMD again. What will Intel do to gain foothold again (that isn't sabotaging the other)? We need to keep the competition going, or the other side may get too comfortable.
- pbsd 2y ago
  A credible but unconfirmed rumor I've read is that Intel didn't want to do it because of the 512-bit shuffles. The E-cores (including those using the upcoming Skymont microarchitecture) are natively 128-bit and already double-pump AVX2, so to quad-pump AVX-512 would result in many-uops 512-bit shuffle instructions made out of 128-bit uops. This would render the shuffles unusable, because you'd unpredictably have them costing either 1 uop or cycle to taking 10-30 uops/cycles depending on which core you are on at the moment. A situation similar to PEXT/PDEP, which cost almost nothing on Intel and dozens of cycles on AMD until a couple generations ago. Why does Zen 4 not have this problem? First, they're only double-pumping instead of quad-pumping. Secondly, while most of their AVX-512 implementation is double-pumped, there seems to be a full-length shuffle unit in there.
  - Aardwolf 2y ago
    Interesting stuff! If a shuffle is just crossed wires without any logic (as far as I know shuffle is a completely predetermined bit permutation), wouldn't it fit on the E-cores easily enough?
neonsunset 2y ago
Zen 4 AVX512 implementation is not double-pumped and tech journos need to stop calling it that, because it has specific meaning that does not match what takes place. It simply decodes operations on ZMM registers into multiple uOPS and schedules them to free 256b units. In addition, Zen 4 has special handling of 512b full-width shuffles, with dedicated hardware to avoid doing very expensive emulation. As a result, Zen 4 with its 4 256b SIMD units still acts like a very strong 2x512b core. There is nothing cheap about this implementation and it is probably the best rendition for consumer hardware so far.

Tuna-Fish 2y ago

If Intel actually implements AVX10/256 on every CPU they ship going forwards, it will eventually win simply by availability. The market has repeatedly and thoroughly rejected dispatching to different code paths based on CPU, so the only SIMD implementation that actually matters is the lowest common denominator. And since AVX10.1/256 and AVX512VL have a shared subset, that will be what people will eventually target once enough time has passed and nearly everyone has a CPU that can support it. AVX512 will continue to give AMD some easy wins on the few benchmarking apps that were actually updated to support it, but if Intel sticks with the AVX10 plan I expect that AMD will eventually just use the double-pumped SIMD pipes for everything, just because they are the more efficient way to support AVX10/256 while retaining AVX512 compatibility. Intel did a lot of bad choices in the past decade, but segmenting the market based on instruction set has to be one of the worst. They just chose to kill all the momentum and interest in their newest and best innovations. Hopefully they actually add AVX10/256 support to the whole lineup, because the width is the least interesting part about AVX512, the masked operations especially are a lot more important.

Remnant44 2y ago
I agree. I still hesitate to ship code that requires even AVX1, even though it was first introduced in 2011(!). AVX512 really improves the instruction set. Not just from masking but from some really big holes filled in terms of instructions available that AVX2 doesn't have a good solution for. At the same time I also DO have plenty of code that could definitely use the compute throughput improvement of 512 bit vectors. But it's definitely a more niche usage. You have to at least nominally satisfy that you: 1) Benefit from 2x the ALU throughput 2) Live mostly in the cache 3) Are not worth running on the GPU instead.
- Const-me 2y ago
  It depends on the product and the market. I’ve been shipping software that requires AVX2 and FMA3 because it is a professional CAM/CAE application. Our users typically don’t use sub-$100 Intel processors like Pentium and Celeron. The key is communication, you need to make sure users are aware of the hardware requirements before they buy. Another example, in the server market, you can almost always count on having at least AVX2 support because most servers today are cloud-based. For people running these cloud services, power efficiency is crucial due to the high cost of electricity, so they tend to replace old hardware regularly. On the other hand, for desktop software aimed at a broad audience, there are valid reasons to hesitate before shipping code that requires AVX1.
janwas 2y ago
Dispatching is actually heavily used. Image/video codecs, cryptography, and ML libraries routinely use it, because the lowest common denominator is very low indeed.
- Tuna-Fish 2y ago
  The things you listed are probably less than 1% of all client loads. Video codecs and ML mostly run on accelerators and cryptography is a tiny proportion of all loads.
  - richardwhiuk 2y ago
    Surely h264 encode and decode is substantial, given the large amount of video consumed?
    - Const-me 2y ago
      I don’t believe many people encode or especially decode video with CPU-running codes. Modern GPUs include hardware accelerators for popular video codecs, these are typically way more power efficient than even AVX-512 software implementations.
      - account42 2y ago
        > I don’t believe many people encode or especially decode video with CPU-running codes. Believe what you want but as soon as realtime is not a concern and quality matters you'll be using CPU-based encoders unless you have special hardware for your use case. > Modern GPUs include hardware accelerators for popular video codecs ... with shit quality encoders because they are designed for speed first. Decoding is a different matter but even there older hardware can easily end up not supporting codecs (or profiles of them) that you come accross.
  - zekica 2y ago
    Cryptography is every download you ever do in a browser, bitlocker or equivalent encrypted disk access. Add to that gzip or brotli compression (common in HTTP). Hardware decoders for newer video codecs (such as AV1) are not that common (less than 50% of devices) so most youtube watch time on desktops/laptops is software decoding. It's a lot more than 1%.
School-Cotton 2y ago
> The market has repeatedly and thoroughly rejected dispatching to different code paths based on CPU What do you mean? At least numpy and pytorch (the only numeric libraries I'm familiar with) both use runtime dispatching.
nemetroid 2y ago
> The market has repeatedly and thoroughly rejected dispatching to different code paths based on CPU, so the only SIMD implementation that actually matters is the lowest common denominator. RHEL just moved up to x86_64-v2, equivalent to 2009 level CPUs. And they’re an early mover, Fedora/Ubuntu/Arch have not done the same. Glibc has used CPU dispatching for str* and mem* functions for over a decade.
- IWeldMelons 2y ago
  FreeBSD added AVX512 dispatch into it C library AFAIK.
TinkersW 2y ago
I don't get saying mask operations are more important than width? Mask operations can be trivially emulated with vblend, it is one extra instruction.. Width can't be emulated, you just are stuck running half speed. This take keeps getting repeated, but doesn't appear to be backed up by reality. Intel hasn't even put AVX10 on their upcoming chips(skymont), so it appears to be going nowhere.
- account42 2y ago
  > Mask operations can be trivially emulated with vblend, it is one extra instruction.. For unaligned loads where you can't guarantee that the entire vector is on a mapped page?
- fanf2 2y ago
  The important feature of AVX-512 demonstrated in my blog post is masked loads and stores, which can't be emulated with vblend.

Aardwolf 2y ago

I don't understand why intel doesn't make their E-cores use double pumped AVX512 to solve this issue (or just make P-core only CPUs for desktops like it should). They have had years to fix this by now. It's annoying that despite AMD's support, market share makes the adoption of this not happen anyway, and the AVX10 thing will unfortunately allow them to hold back the world even longer. What I like to see (for desktop) is: better cores, more cores, well standardized instruction sets that unlock useful things (wide SIMD, float16, gather/scatter, ...). AMD is doing pretty well at this. What Intel is doing instead: put weak cores alongside decent cores, cripple the decent cores to keep up with the weak cores, release CPUs with the same amount of cores as before for many generations in a row, use the weak cores to make it sound like they have more cores than they have, bring out way too many variants of instructions sets to ever have a useful common set, drop support for their own promising sounding instructions I just really dislike anything Intel has come out with or plans to come out with lately :p My cycle of preference of manufacturer (for desktop computers) has been: 90s: Intel. Early 2000s: AMD (pentium 4 was so meh). late 2000's+2010s: Intel. Now: AMD again. What will Intel do to gain foothold again (that isn't sabotaging the other)? We need to keep the competition going, or the other side may get too comfortable.

pbsd 2y ago
A credible but unconfirmed rumor I've read is that Intel didn't want to do it because of the 512-bit shuffles. The E-cores (including those using the upcoming Skymont microarchitecture) are natively 128-bit and already double-pump AVX2, so to quad-pump AVX-512 would result in many-uops 512-bit shuffle instructions made out of 128-bit uops. This would render the shuffles unusable, because you'd unpredictably have them costing either 1 uop or cycle to taking 10-30 uops/cycles depending on which core you are on at the moment. A situation similar to PEXT/PDEP, which cost almost nothing on Intel and dozens of cycles on AMD until a couple generations ago. Why does Zen 4 not have this problem? First, they're only double-pumping instead of quad-pumping. Secondly, while most of their AVX-512 implementation is double-pumped, there seems to be a full-length shuffle unit in there.
- Aardwolf 2y ago
  Interesting stuff! If a shuffle is just crossed wires without any logic (as far as I know shuffle is a completely predetermined bit permutation), wouldn't it fit on the E-cores easily enough?

neonsunset 2y ago

Zen 4 AVX512 implementation is not double-pumped and tech journos need to stop calling it that, because it has specific meaning that does not match what takes place. It simply decodes operations on ZMM registers into multiple uOPS and schedules them to free 256b units. In addition, Zen 4 has special handling of 512b full-width shuffles, with dedicated hardware to avoid doing very expensive emulation. As a result, Zen 4 with its 4 256b SIMD units still acts like a very strong 2x512b core. There is nothing cheap about this implementation and it is probably the best rendition for consumer hardware so far.