10 ms·

Intel AVX10: The Successor to AVX-512

97 points by gautamcgoel 3y ago

Am4TIfIsER0ppos 3y ago
Slow down dammit! I've barely started writing avx512 since they became worth it on ice lake. > being able to work for both P and E cores Oh yes I forgot they were gimping their own processors. > the converged version has a maximum vector length of 256-bits [on] the E cores while P cores will have optional 512-bit vector use Maybe they shouldn't have made xmm and ymm "extensions" to the base set to begin with.
- eklitzke 3y ago
  Extension in this context basically just means new registers/instructions that didn't exist on a 386. Even floating point support on x86 is technically an extension. SSE (which adds the xmm registers) has existed on all x86 CPUs made by Intel in the last 20+ years, and the ymm registers/instructions are part of AVX2 which has existed on all x86 CPUs made by Intel in the last 10+ years.
  - Am4TIfIsER0ppos 3y ago
    I don't mean mmx through avx2 when I said extensions. I mean the "vector length" extensions/flag for avx512 that lets you know you can use those instructions with xmm and ymm sized operands. Since it is a flag it means it is a theoretical optional feature whereas it should have just been part of the avx512 base. With the hindsight of what Intel want for avx10 there should have been flags for xmm, ymm, and zmm operands. That would be excellent for the "E" cores.
canucker2016 3y ago
from https://cdrdv2.intel.com/v1/dl/getContent/784343 https://cdrdv2.intel.com/v1/dl/getContent/784343 ("The Converged Vector ISA: Intel Advanced Vector Extensions 10" Technical Paper PDF) "Intel AVX10 Version 1 will be introduced for early software enablement and supports the subset of all the Intel AVX-512 instruction set available as of future Intel Xeon processors with P-cores, codenamed Granite Rapids, that is forward compatible to Intel AVX10. This version will not include the new 256-bit vector instructions supporting embedded rounding or any of the new instructions and will serve as the transition base version from Intel AVX-512 to Intel AVX10. Intel AVX10 Version 2 will include the 256-bit instruction forms supporting embedded rounding as well as a suite of new Intel AVX10 instructions covering new AI data types and conversions, data movement optimizations, and standards support. All new instructions will be supported at 128-, 256-, and 512-bit vector lengths with limited variances. All Intel AVX10 versions will implement the new versioning enumeration scheme." And who knows when AMD will have time to update Zen ? architecture with these new instructions.
- jiggawatts 3y ago
  The latest shipping processor is the 4-th generation Intel Xeon "Sapphire Rapids". AVX10 looks like it won't be fully supported until the 6-th generation "Granite Rapids" Xeons, probably in 2025. So AMD has plenty of time to implement the additional instructions, of which there don't appear to be that many. Mostly it's a unification of feature sets and a simplification of the CPU ID flags. Also, Intel is still nerfing the efficiency "E" cores to AVX10-256, so the instruction sets will remain non-uniform in practice for the foreseeable future. Meanwhile, AMD is implementing AVX-512 on all of their CPU cores including in the high-density 128-core AMD EPYC "Bergamo" line, which is a significant differentiator. Intel just cannot resist segmenting a market, even when announcing that they're ending market segmentation!
  - brucethemoose2 3y ago
    > So AMD has plenty of time to implement the additional instructions Architectures take years to plan. If AMD is just finding out, I'm not sure they have time. However, one would hope AMD had a little heads up. Its in Intel's and AMD's best interest not to flub the next AVX, with ARM SVE2 (and who knows what else) around the corner.
colejohnson66 3y ago
That’s a massive extension. 32 GPRs! And they’re finally reusing an encoding made reserved in long mode (D5 - AAM in legacy modes). Guess Intel’s feeling the pressure from Zen 4 supporting AVX-512.
- Am4TIfIsER0ppos 3y ago
  Where did you see 32 general purpose registers? There are 32 vector registers in avx512 and avx10. Did I miss it?
  - arthur2e5 3y ago
    32 GPR is a different extension, APX (Adv. Performance Ext.), which also has arm-ish predicated insturctions.
crest 3y ago
Hasn't AMD proven multiple times that a double pumped packed-SIMD implementation works well enough? Just the permute operations need a full width data path to get reasonable latencies. Intel already overplayed their hand with AVX-512 when they still had a stronger position. Let's hope they fail to hold back the field with their misguided attempts to increase their margin no matter the cost (even to their own bottom line).
- mort96 3y ago
  My understanding is that, while double pumped 512 bit makes sense from AMD from a compatibility perspective, it's not a great solution. The sheer amount of architectural stare you need to keep track of with all these 512 bit instructions gets unwieldy, especially on smaller cores, and there's no real performance advantage to double pumped 512 vs normal 256. This new ISA seems to include 256 bit versions of the instructions which were previously only in AVX-512 though. That's exciting.
  - adrian_b 3y ago
    Actually on Zen 4 using 512-bit instructions instead of 256-bit instructions frequently improves the performance a lot, despite using the same execution units. The reason is that in Zen 4 the relatively narrow instruction decoder is frequently the bottleneck, so twice less instructions can influence the speed very much. On future CPUs that will be able to decode more instructions in parallel, like Zen 5 and the future Intel CPUs, 512-bit instructions might no longer increase the speed when the width of the execution units remains of 256 bits, so perhaps even AMD may introduce a split between laptop CPUs that implement only the 256-bit subset of AVX-512 rebranded AVX10 and server CPUs that implement the full AVX-512 a.k.a. AVX10.
  - Dylan16807 3y ago
    The architectural state is quadrupled, but the real impact is much smaller. Single-pumped AVX2 implementations already had ~150 extra 256-bit registers for pipelining purposes. And if you're double pumping 512-bit instructions, then you can pipeline just as deep without changing the quantity or width. Maintaining similar performance is not 512B->2KB, it's 5KB->7KB. (But note that the E cores are largely double-pumping 256 bit instructions already, so for them it's a big change.)
dathinab 3y ago
Is someone here who understands the nitty bitty details of AVX-512/AVX10 and could tell me what is included which current latest gen AMD processors do not support? Because the only thing I can pick out is the 256bit AVX-512 which AFIK recent amd processors do support (including 512bit support) both on their normal cores and their new compacted code. But I don't know much about AVX_ so I'm 100% I missed a bunch of stuff and/or limitations with current AMD code (besides it being double pumped).
- zokier 3y ago
  Afaik the thing Zen is missing currently is FP16.
- jcranmer 3y ago
  > Is someone here who understands the nitty bitty details of AVX-512/AVX10 and could tell me what is included which current latest gen AMD processors do not support? Half-precision floating-point for sure.
  - adrian_b 3y ago
    As another poster has already mentioned, Zen 4 supports one half-precision FP format, BF16, which was first introduced in Intel Cooper Lake. Sapphire Rapids has added an alternative format FP16, which is not supported by Zen 4, because Zen 4 was launched before Sapphire Rapids.
- Falvyu 3y ago
  Zen 4 lacks AVX512_FP16 (for 16-bits IEEE floating point operations), AVX512_VP2INTERSECT and also lack the Advanced Matrix eXtension (AMX) set (if you consider that part of AVX512). https://twitter.com/InstLatX64/status/1646471371558461445/ https://twitter.com/InstLatX64/status/1646471371558461445/ It's worth noting that 16-bits floating point operations are still possible on Zen 4, but using Google's BF16.
- Zandikar 3y ago
  AVX512 subsets *NOT* supported on AMD Zen4 (Epyc 9000, Ryzen 7000): - AVX512ERAVX512PF (Knights Landing) - AVX512 4VNNIW, 4FMAPS (Knights Mill) - VP2INTERSECT (Tiger Lake) - FP16 (Cooper Lake) You can see a table here showing cpu's AVX512 support[0], and wikicpu also has that data in a list [1], along with more detailed info. It's worth noting all data I can find specifically talks about AVX512 support in terms of the architecture (Zen4), not anything specific product families such as ryzen vs epyc implementations. It's possible there is/will be some artificial product segmentation going on (like what Intel did retroactively with Alder Lake to disable AVX512) but I've not seen any mention of it regarding AMD's products. I mention this because Intel has kinda been all over the place with AVX512 implementation, and there's a big difference between Enterprise and Consumer in how they deliver/treat AVX512. AMD obviously is a different company with different practices, but with only 1 generation to go off of it's hard to speculate on a trend, and AVX512 can be an expensive implementation. Point being: It's unfortunately complicated AND subject to change. The "big" one here for ML/DL/AI vs Intel's Sapphire Rapids (and presumably going forward) is FP16 not being supported on AMD. Of course, it's contextual as to whether that's relevant to you though. [0]: https://en.wikipedia.org/wiki/AVX-512?oldformat=true#CPUs_with_AVX-512 https://en.wikipedia.org/wiki/AVX-512?oldformat=true#CPUs_wi... [1]: https://en.wikichip.org/wiki/amd/microarchitectures/zen_4#New_Instructions https://en.wikichip.org/wiki/amd/microarchitectures/zen_4#Ne... My experience is primarily with Intel's implementation of AVX512 in their Xeon products for DL/AI. I can expand on that if interested, but unfortunately haven't had a chance to play with AMD Zen4 yet in that domain, and it's gonna be a while till I can get hands on with a Epyc 9000 probably. EDIT: Fixed typos
phkahler 3y ago
Can we get great RISC-V cores from Apple or AMD please with that vector ISA so we can shut down this whole notion of ISA as a product differentiator?
- camel-cdr 3y ago
  Intel horse creek, sadly won't include the vector isa, just RV64GBC.
- dragontamer 3y ago
  One of the reasons to use Intel is that their L1 cache is sized to 512-bit operations. AMD Zen4 has 256-bit load/store to L1 cache, which means it'd take a load/store unit 2-clock ticks to execute any 512-bit load or store. (Though Zen4 has multiple load/store units, so maybe its not that big of a deal in practice). ------------- GPUs are 32-wide with 32-bits each, or 1024-wide operations in practice. GCN was 64-wide with 32-bits each or 2048 wide in practice. The reason why AVX512 was a big deal isn't the width btw. But instead the design of the instruction set. AVX512 is the most advanced CPU vector instruction set ever made, with huge inspirations from NVidia's PTX or AMD's GCN. Of course, GPU-assembly remains superior. I don't think that RISC-V (or whoever) should be copying AVX512 or even AVX10, they should be looking at NVidia PTX or NVidia SASS as inspiration.
  - camel-cdr 3y ago
    > AVX512 is the most advanced CPU vector instruction set ever made, with huge inspirations from NVidia's PTX or AMD's GCN. Can you explain what makes it the above? I haven't really worked with avx512 before, only up to avx2 and rvv.
    - dragontamer 3y ago
      1. AVX512 allows a 64-bit conditional mask to be used over the 64-bytes (aka a 512-bit operation). Like GPUs (NVidia or AMD) before it, this allows for far simpler implementations of if-statements (or loops) in practice. 2. AVX512 has vgather and vscatter, aka vector-load and vector-store. While no computer accelerates this process, this still simplifies programming and read/writes to memory can be thought of on a per-lane basis. 3. Compress and Expand, new instructions not found in GPU-space, perform the stream-compression operation (and inverse operation). See: https://www.cse.chalmers.se/~uffe/streamcompaction.pdf https://www.cse.chalmers.se/~uffe/streamcompaction.pdf , lots of practical applications here, nice to see just a simple "instruction" that handles this detail. Though GPU programmers can prefix-sum and build this primitive themselves in practice, its a common enough operation that a proper-instruction is likely a good idea. ------------ Just a few tidbits of AVX512 ISA that shows that Intel really knows what they're doing here. #1 can be seen as a 64-bit generalization to the cmov conditional instructions that are in Intel's normal 64-bit instruction set. Except instead of being a "1-bit" compare or 1-bit if-statement, might as well generalize it to a 64-bit operation across 64x8-bit bytes that naturally map into a 512-bit register, right? Its "why" you can just say "if(blahblah)" in CUDA or OpenCL, because under-the-hood, the GPUs are holding onto these 64-bit conditional registers and are turning-off and turning-on lanes to execute. (GPUs don't do if(){A();} else () {B();} like a CPU. GPUs turn on conditional-execution masks per-lane and then execute both A-and-B, though if the lane is "off" then the results are thrown away). It all comes down to S-registers (GCN), opcode-masks (AVX512) or whatnot, its fundamental to "proper" GPU-like SIMD. In any case, AVX512 is still playing catchup to the gpu space, but Intel has shown me (IMO) that they're the CPU-assembly group that has fully understood what makes GPUs tick and why parallelism is possible in CUDA / OpenCL. Maybe they'll leapfrog the GPU guys in the next few years, but I still consider GPUs to be the cream-of-the-crop in terms of assembly-language design for parallel SIMD systems... for today anyway. Then again, NVidia has also shown themselves to be very innovative in this low-level space. And in many ways, its easier to "catch up" (as Intel has done) rather than trailblaze. Still, Intel showing off compression instructions is a good sign, it is true innovation.
jauntywundrkind 3y ago
This seemed really cool. I'm used to a lot of new instructions & boosts, but Intel adding new conditional load/store is a smart interesting coupling that could help increase execution unit efficiency in a significant way. > As out-of-order CPUs continue to become deeper and wider, the cost of mispredictions increasingly dominates performance of such workloads. Branch predictor improvements can mitigate this to a limited extent only as data-dependent branches are fundamentally hard to predict. > To address this growing performance issue, we significantly expand the conditional instruction set of x86, which was first introduced with the Intel® Pentium® Pro in the form of CMOV/SET instructions. These instructions are used quite extensively by today’s compilers, but they are too limited for broader use of if-conversion (a compiler optimization that replaces branches with conditional instructions). > Intel® APX adds conditional forms of load, store, and compare/test instructions, and it also adds an option for the compiler to suppress the status flags writes of common instructions. https://www.intel.com/content/www/us/en/developer/articles/technical/advanced-performance-extensions-apx.html https://www.intel.com/content/www/us/en/developer/articles/t... I didn't understand everything about the "caller-saved volatile" new general purpose register interface & legacy compatibility. But some potentially really interesting optimizations where load/store being dual register capable, and being capable of staying on the AVX unit & not having to go further out to "memory" (caches?): > Generally, more register state will need to be managed at function boundaries. In order to reduce the associated overhead, we are adding PUSH2/POP2 instructions that transfer two register values within a single memory operation. The processor tracks these new instructions internally and fast-forwards register data between matching PUSH2 and POP2 instructions without going through memory. Neat stuff. Very superficially reminds me of Semantic Streaming Registers on the very novel standalone-ish FPU on PULP's RISC-V based Occamy many-core chip. In that the unit is acting in a more standalone fashion. https://www.youtube.com/watch?v=kMhdq7A3d3I#t=10m https://www.youtube.com/watch?v=kMhdq7A3d3I#t=10m https://pulp-platform.org/docs/BeniniSC11-22.pdf https://pulp-platform.org/docs/BeniniSC11-22.pdf
- adrian_b 3y ago
  > I didn't understand everything about the "caller-saved volatile" Whenever new registers are added to a CPU, it may be necessary to modify the operating system in order to save the new registers across context switches, but it is possible to change the ABI (i.e. the conventions for calling functions) in such a way that will allow the new functions that use the new registers to be linked with old functions that are not aware of the existence of the new registers. To achieve this goal, it is enough to define in the ABI all the new registers as "caller-saved volatile". This means that any function that uses some of the new registers must not assume anything about the values of those registers upon function entry and it must save them before calling any other function. This ensures that regardless what other functions do, intentionally or accidentally, the values stored in the new registers cannot be corrupted, so it is safe to mix new functions that are aware of them with old functions that ignore them. Had the new registers been defined e.g. as registers that must be preserved across calls, i.e. which must be saved inside the invoked function, that would not have worked, because an old function would not have known how to save them.
shmerl 3y ago
Why did it take around 10 years for AMD to implement AVX-512 and will they need to wait as long for this too? Doesn't seem to be patent related (patents are 20 years and AVX-512 was introduced in 2013?).
- meepmorp 3y ago
  Don’t Intel and AMD have cross licensing agreements for instruction set extensions?
  - shmerl 3y ago
    That's what I thought too. But somehow AVX-512 took a long time for AMD.
    - colejohnson66 3y ago
      Implementing an instruction set isn’t as easy as “we have access to IP”
- dralley 3y ago
  Because Intel implementations of AVX-512 were pretty much unusable or not worth using until 2019-2020, so nobody was writing much AVX-512, so they didn't need to. AMD practically resurrected AVX-512 by supporting it across both desktop and server products, so it's weird to criticize them for taking a while when Intel nearly killed the thing via segmentation and poor implementation.
  - shmerl 3y ago
    I'm not criticizing it, just wondering why it took that long and if it was due to patents or not.
    - danzk 3y ago
      AMD probably decided that their transistor budget was better spent elsewhere.
codedokode 3y ago
From the name I thought that they extended registers to 1024 bits, but it looks like instead they made 512-bit width support optional.
- imtringued 3y ago
  Reminds me of OpenCL 3.0...
RcouF1uZ4gsC 3y ago
It seems Intel is borrowing Microsoft’s XBox naming scheme. At least they didn’t name the successor to AVX-512 AVX-One.
aperture147 3y ago
As an average developer who works on high level interface, I don't really see the benefit of AVX-512. I've heard that some math calculating software MAY gains some benefit from AVX instructions (like BLAS), but I've never use it personally. Can you guys please explain?
- physicsguy 3y ago
  Vector instructions speed up things like: for (int i = 0; i < N; i++) { a[i] = b[i]*c[i]; } Effectively, they let you do this in chunks of 2, 4 or 8 elements in a single instruction instead of element by element. Usually the clock speed of the processor drops slightly while executing these instructions but not so much as to make it slower than not having the instruction. New instruction sets usually do two things - add support for new operations, and widen the registers and allow you to perform the operation on more elements at a time. If you’re writing in a high level language, you’ll only really see performance improvements if your interpreter or library takes advantage of them. For e.g. the Python library NumPy makes good use of vectorisation. In terms of writing low level code, simple loops are often auto-vectorised by the compiler so you often see a sort of odd style where a loop doing many things is split up so the compiler can deal with it. You often end up having to run the code through something like VTune to work out whether a particular loop has actually vectorised.
- crote 3y ago
  Anything which works on nontrivial datasets, really. As soon as you need to do roughly the same operation on a bunch of data, you can benefit from AVX. Heck, this could be as simply as determining the length of a string! The main benefit of AVX-512 is that the CPU gained support for "masking". Traditionally, you had to execute the instructions on all data elements. Something like "round all even values downwards, round all odd values upwards" becomes near-impossible, and if you want to do that in an entire AVX pipeline you have to deconstruct the vector into individual elements, do the operation on each of them individually, and reconstruct it into a vector. This really sucks. With masking, you get a special field which specify on which elements the operation should apply. So you could just create a mask with the even values, use that mask for a round-down, invert the mask, and use it for a round-up. That's a significant speedup! Adding AVX-512 support means that a lot more applications suddenly become eligible for fairly trivial vectorization, and I personally can't wait for it to become universally available.

97 points by gautamcgoel 3y ago

Slow down dammit! I've barely started writing avx512 since they became worth it on ice lake. > being able to work for both P and E cores Oh yes I forgot they were gimping their own processors. > the converged version has a maximum vector length of 256-bits [on] the E cores while P cores will have optional 512-bit vector use Maybe they shouldn't have made xmm and ymm "extensions" to the base set to begin with.

eklitzke 3y ago
Extension in this context basically just means new registers/instructions that didn't exist on a 386. Even floating point support on x86 is technically an extension. SSE (which adds the xmm registers) has existed on all x86 CPUs made by Intel in the last 20+ years, and the ymm registers/instructions are part of AVX2 which has existed on all x86 CPUs made by Intel in the last 10+ years.
- Am4TIfIsER0ppos 3y ago
  I don't mean mmx through avx2 when I said extensions. I mean the "vector length" extensions/flag for avx512 that lets you know you can use those instructions with xmm and ymm sized operands. Since it is a flag it means it is a theoretical optional feature whereas it should have just been part of the avx512 base. With the hindsight of what Intel want for avx10 there should have been flags for xmm, ymm, and zmm operands. That would be excellent for the "E" cores.

canucker2016 3y ago

from https://cdrdv2.intel.com/v1/dl/getContent/784343 https://cdrdv2.intel.com/v1/dl/getContent/784343 ("The Converged Vector ISA: Intel Advanced Vector Extensions 10" Technical Paper PDF) "Intel AVX10 Version 1 will be introduced for early software enablement and supports the subset of all the Intel AVX-512 instruction set available as of future Intel Xeon processors with P-cores, codenamed Granite Rapids, that is forward compatible to Intel AVX10. This version will not include the new 256-bit vector instructions supporting embedded rounding or any of the new instructions and will serve as the transition base version from Intel AVX-512 to Intel AVX10. Intel AVX10 Version 2 will include the 256-bit instruction forms supporting embedded rounding as well as a suite of new Intel AVX10 instructions covering new AI data types and conversions, data movement optimizations, and standards support. All new instructions will be supported at 128-, 256-, and 512-bit vector lengths with limited variances. All Intel AVX10 versions will implement the new versioning enumeration scheme." And who knows when AMD will have time to update Zen ? architecture with these new instructions.

jiggawatts 3y ago
The latest shipping processor is the 4-th generation Intel Xeon "Sapphire Rapids". AVX10 looks like it won't be fully supported until the 6-th generation "Granite Rapids" Xeons, probably in 2025. So AMD has plenty of time to implement the additional instructions, of which there don't appear to be that many. Mostly it's a unification of feature sets and a simplification of the CPU ID flags. Also, Intel is still nerfing the efficiency "E" cores to AVX10-256, so the instruction sets will remain non-uniform in practice for the foreseeable future. Meanwhile, AMD is implementing AVX-512 on all of their CPU cores including in the high-density 128-core AMD EPYC "Bergamo" line, which is a significant differentiator. Intel just cannot resist segmenting a market, even when announcing that they're ending market segmentation!
- brucethemoose2 3y ago
  > So AMD has plenty of time to implement the additional instructions Architectures take years to plan. If AMD is just finding out, I'm not sure they have time. However, one would hope AMD had a little heads up. Its in Intel's and AMD's best interest not to flub the next AVX, with ARM SVE2 (and who knows what else) around the corner.

colejohnson66 3y ago

That’s a massive extension. 32 GPRs! And they’re finally reusing an encoding made reserved in long mode (D5 - AAM in legacy modes). Guess Intel’s feeling the pressure from Zen 4 supporting AVX-512.

Am4TIfIsER0ppos 3y ago
Where did you see 32 general purpose registers? There are 32 vector registers in avx512 and avx10. Did I miss it?
- arthur2e5 3y ago
  32 GPR is a different extension, APX (Adv. Performance Ext.), which also has arm-ish predicated insturctions.

crest 3y ago

Hasn't AMD proven multiple times that a double pumped packed-SIMD implementation works well enough? Just the permute operations need a full width data path to get reasonable latencies. Intel already overplayed their hand with AVX-512 when they still had a stronger position. Let's hope they fail to hold back the field with their misguided attempts to increase their margin no matter the cost (even to their own bottom line).

mort96 3y ago
My understanding is that, while double pumped 512 bit makes sense from AMD from a compatibility perspective, it's not a great solution. The sheer amount of architectural stare you need to keep track of with all these 512 bit instructions gets unwieldy, especially on smaller cores, and there's no real performance advantage to double pumped 512 vs normal 256. This new ISA seems to include 256 bit versions of the instructions which were previously only in AVX-512 though. That's exciting.
- adrian_b 3y ago
  Actually on Zen 4 using 512-bit instructions instead of 256-bit instructions frequently improves the performance a lot, despite using the same execution units. The reason is that in Zen 4 the relatively narrow instruction decoder is frequently the bottleneck, so twice less instructions can influence the speed very much. On future CPUs that will be able to decode more instructions in parallel, like Zen 5 and the future Intel CPUs, 512-bit instructions might no longer increase the speed when the width of the execution units remains of 256 bits, so perhaps even AMD may introduce a split between laptop CPUs that implement only the 256-bit subset of AVX-512 rebranded AVX10 and server CPUs that implement the full AVX-512 a.k.a. AVX10.
- Dylan16807 3y ago
  The architectural state is quadrupled, but the real impact is much smaller. Single-pumped AVX2 implementations already had ~150 extra 256-bit registers for pipelining purposes. And if you're double pumping 512-bit instructions, then you can pipeline just as deep without changing the quantity or width. Maintaining similar performance is not 512B->2KB, it's 5KB->7KB. (But note that the E cores are largely double-pumping 256 bit instructions already, so for them it's a big change.)

dathinab 3y ago

Is someone here who understands the nitty bitty details of AVX-512/AVX10 and could tell me what is included which current latest gen AMD processors do not support? Because the only thing I can pick out is the 256bit AVX-512 which AFIK recent amd processors do support (including 512bit support) both on their normal cores and their new compacted code. But I don't know much about AVX_ so I'm 100% I missed a bunch of stuff and/or limitations with current AMD code (besides it being double pumped).

zokier 3y ago
Afaik the thing Zen is missing currently is FP16.
jcranmer 3y ago
> Is someone here who understands the nitty bitty details of AVX-512/AVX10 and could tell me what is included which current latest gen AMD processors do not support? Half-precision floating-point for sure.
- adrian_b 3y ago
  As another poster has already mentioned, Zen 4 supports one half-precision FP format, BF16, which was first introduced in Intel Cooper Lake. Sapphire Rapids has added an alternative format FP16, which is not supported by Zen 4, because Zen 4 was launched before Sapphire Rapids.
Falvyu 3y ago
Zen 4 lacks AVX512_FP16 (for 16-bits IEEE floating point operations), AVX512_VP2INTERSECT and also lack the Advanced Matrix eXtension (AMX) set (if you consider that part of AVX512). https://twitter.com/InstLatX64/status/1646471371558461445/ https://twitter.com/InstLatX64/status/1646471371558461445/ It's worth noting that 16-bits floating point operations are still possible on Zen 4, but using Google's BF16.
Zandikar 3y ago
AVX512 subsets *NOT* supported on AMD Zen4 (Epyc 9000, Ryzen 7000): - AVX512ERAVX512PF (Knights Landing) - AVX512 4VNNIW, 4FMAPS (Knights Mill) - VP2INTERSECT (Tiger Lake) - FP16 (Cooper Lake) You can see a table here showing cpu's AVX512 support[0], and wikicpu also has that data in a list [1], along with more detailed info. It's worth noting all data I can find specifically talks about AVX512 support in terms of the architecture (Zen4), not anything specific product families such as ryzen vs epyc implementations. It's possible there is/will be some artificial product segmentation going on (like what Intel did retroactively with Alder Lake to disable AVX512) but I've not seen any mention of it regarding AMD's products. I mention this because Intel has kinda been all over the place with AVX512 implementation, and there's a big difference between Enterprise and Consumer in how they deliver/treat AVX512. AMD obviously is a different company with different practices, but with only 1 generation to go off of it's hard to speculate on a trend, and AVX512 can be an expensive implementation. Point being: It's unfortunately complicated AND subject to change. The "big" one here for ML/DL/AI vs Intel's Sapphire Rapids (and presumably going forward) is FP16 not being supported on AMD. Of course, it's contextual as to whether that's relevant to you though. [0]: https://en.wikipedia.org/wiki/AVX-512?oldformat=true#CPUs_with_AVX-512 https://en.wikipedia.org/wiki/AVX-512?oldformat=true#CPUs_wi... [1]: https://en.wikichip.org/wiki/amd/microarchitectures/zen_4#New_Instructions https://en.wikichip.org/wiki/amd/microarchitectures/zen_4#Ne... My experience is primarily with Intel's implementation of AVX512 in their Xeon products for DL/AI. I can expand on that if interested, but unfortunately haven't had a chance to play with AMD Zen4 yet in that domain, and it's gonna be a while till I can get hands on with a Epyc 9000 probably. EDIT: Fixed typos

phkahler 3y ago

Can we get great RISC-V cores from Apple or AMD please with that vector ISA so we can shut down this whole notion of ISA as a product differentiator?

camel-cdr 3y ago
Intel horse creek, sadly won't include the vector isa, just RV64GBC.
dragontamer 3y ago
One of the reasons to use Intel is that their L1 cache is sized to 512-bit operations. AMD Zen4 has 256-bit load/store to L1 cache, which means it'd take a load/store unit 2-clock ticks to execute any 512-bit load or store. (Though Zen4 has multiple load/store units, so maybe its not that big of a deal in practice). ------------- GPUs are 32-wide with 32-bits each, or 1024-wide operations in practice. GCN was 64-wide with 32-bits each or 2048 wide in practice. The reason why AVX512 was a big deal isn't the width btw. But instead the design of the instruction set. AVX512 is the most advanced CPU vector instruction set ever made, with huge inspirations from NVidia's PTX or AMD's GCN. Of course, GPU-assembly remains superior. I don't think that RISC-V (or whoever) should be copying AVX512 or even AVX10, they should be looking at NVidia PTX or NVidia SASS as inspiration.
- camel-cdr 3y ago
  > AVX512 is the most advanced CPU vector instruction set ever made, with huge inspirations from NVidia's PTX or AMD's GCN. Can you explain what makes it the above? I haven't really worked with avx512 before, only up to avx2 and rvv.
  - dragontamer 3y ago
    1. AVX512 allows a 64-bit conditional mask to be used over the 64-bytes (aka a 512-bit operation). Like GPUs (NVidia or AMD) before it, this allows for far simpler implementations of if-statements (or loops) in practice. 2. AVX512 has vgather and vscatter, aka vector-load and vector-store. While no computer accelerates this process, this still simplifies programming and read/writes to memory can be thought of on a per-lane basis. 3. Compress and Expand, new instructions not found in GPU-space, perform the stream-compression operation (and inverse operation). See: https://www.cse.chalmers.se/~uffe/streamcompaction.pdf https://www.cse.chalmers.se/~uffe/streamcompaction.pdf , lots of practical applications here, nice to see just a simple "instruction" that handles this detail. Though GPU programmers can prefix-sum and build this primitive themselves in practice, its a common enough operation that a proper-instruction is likely a good idea. ------------ Just a few tidbits of AVX512 ISA that shows that Intel really knows what they're doing here. #1 can be seen as a 64-bit generalization to the cmov conditional instructions that are in Intel's normal 64-bit instruction set. Except instead of being a "1-bit" compare or 1-bit if-statement, might as well generalize it to a 64-bit operation across 64x8-bit bytes that naturally map into a 512-bit register, right? Its "why" you can just say "if(blahblah)" in CUDA or OpenCL, because under-the-hood, the GPUs are holding onto these 64-bit conditional registers and are turning-off and turning-on lanes to execute. (GPUs don't do if(){A();} else () {B();} like a CPU. GPUs turn on conditional-execution masks per-lane and then execute both A-and-B, though if the lane is "off" then the results are thrown away). It all comes down to S-registers (GCN), opcode-masks (AVX512) or whatnot, its fundamental to "proper" GPU-like SIMD. In any case, AVX512 is still playing catchup to the gpu space, but Intel has shown me (IMO) that they're the CPU-assembly group that has fully understood what makes GPUs tick and why parallelism is possible in CUDA / OpenCL. Maybe they'll leapfrog the GPU guys in the next few years, but I still consider GPUs to be the cream-of-the-crop in terms of assembly-language design for parallel SIMD systems... for today anyway. Then again, NVidia has also shown themselves to be very innovative in this low-level space. And in many ways, its easier to "catch up" (as Intel has done) rather than trailblaze. Still, Intel showing off compression instructions is a good sign, it is true innovation.

jauntywundrkind 3y ago

This seemed really cool. I'm used to a lot of new instructions & boosts, but Intel adding new conditional load/store is a smart interesting coupling that could help increase execution unit efficiency in a significant way. > As out-of-order CPUs continue to become deeper and wider, the cost of mispredictions increasingly dominates performance of such workloads. Branch predictor improvements can mitigate this to a limited extent only as data-dependent branches are fundamentally hard to predict. > To address this growing performance issue, we significantly expand the conditional instruction set of x86, which was first introduced with the Intel® Pentium® Pro in the form of CMOV/SET instructions. These instructions are used quite extensively by today’s compilers, but they are too limited for broader use of if-conversion (a compiler optimization that replaces branches with conditional instructions). > Intel® APX adds conditional forms of load, store, and compare/test instructions, and it also adds an option for the compiler to suppress the status flags writes of common instructions. https://www.intel.com/content/www/us/en/developer/articles/technical/advanced-performance-extensions-apx.html https://www.intel.com/content/www/us/en/developer/articles/t... I didn't understand everything about the "caller-saved volatile" new general purpose register interface & legacy compatibility. But some potentially really interesting optimizations where load/store being dual register capable, and being capable of staying on the AVX unit & not having to go further out to "memory" (caches?): > Generally, more register state will need to be managed at function boundaries. In order to reduce the associated overhead, we are adding PUSH2/POP2 instructions that transfer two register values within a single memory operation. The processor tracks these new instructions internally and fast-forwards register data between matching PUSH2 and POP2 instructions without going through memory. Neat stuff. Very superficially reminds me of Semantic Streaming Registers on the very novel standalone-ish FPU on PULP's RISC-V based Occamy many-core chip. In that the unit is acting in a more standalone fashion. https://www.youtube.com/watch?v=kMhdq7A3d3I#t=10m https://www.youtube.com/watch?v=kMhdq7A3d3I#t=10m https://pulp-platform.org/docs/BeniniSC11-22.pdf https://pulp-platform.org/docs/BeniniSC11-22.pdf

adrian_b 3y ago
> I didn't understand everything about the "caller-saved volatile" Whenever new registers are added to a CPU, it may be necessary to modify the operating system in order to save the new registers across context switches, but it is possible to change the ABI (i.e. the conventions for calling functions) in such a way that will allow the new functions that use the new registers to be linked with old functions that are not aware of the existence of the new registers. To achieve this goal, it is enough to define in the ABI all the new registers as "caller-saved volatile". This means that any function that uses some of the new registers must not assume anything about the values of those registers upon function entry and it must save them before calling any other function. This ensures that regardless what other functions do, intentionally or accidentally, the values stored in the new registers cannot be corrupted, so it is safe to mix new functions that are aware of them with old functions that ignore them. Had the new registers been defined e.g. as registers that must be preserved across calls, i.e. which must be saved inside the invoked function, that would not have worked, because an old function would not have known how to save them.

shmerl 3y ago

Why did it take around 10 years for AMD to implement AVX-512 and will they need to wait as long for this too? Doesn't seem to be patent related (patents are 20 years and AVX-512 was introduced in 2013?).

meepmorp 3y ago
Don’t Intel and AMD have cross licensing agreements for instruction set extensions?
- shmerl 3y ago
  That's what I thought too. But somehow AVX-512 took a long time for AMD.
  - colejohnson66 3y ago
    Implementing an instruction set isn’t as easy as “we have access to IP”
dralley 3y ago
Because Intel implementations of AVX-512 were pretty much unusable or not worth using until 2019-2020, so nobody was writing much AVX-512, so they didn't need to. AMD practically resurrected AVX-512 by supporting it across both desktop and server products, so it's weird to criticize them for taking a while when Intel nearly killed the thing via segmentation and poor implementation.
- shmerl 3y ago
  I'm not criticizing it, just wondering why it took that long and if it was due to patents or not.
  - danzk 3y ago
    AMD probably decided that their transistor budget was better spent elsewhere.

codedokode 3y ago

From the name I thought that they extended registers to 1024 bits, but it looks like instead they made 512-bit width support optional.

imtringued 3y ago
Reminds me of OpenCL 3.0...

RcouF1uZ4gsC 3y ago

It seems Intel is borrowing Microsoft’s XBox naming scheme. At least they didn’t name the successor to AVX-512 AVX-One.

aperture147 3y ago

As an average developer who works on high level interface, I don't really see the benefit of AVX-512. I've heard that some math calculating software MAY gains some benefit from AVX instructions (like BLAS), but I've never use it personally. Can you guys please explain?

physicsguy 3y ago
Vector instructions speed up things like: for (int i = 0; i < N; i++) { a[i] = b[i]*c[i]; } Effectively, they let you do this in chunks of 2, 4 or 8 elements in a single instruction instead of element by element. Usually the clock speed of the processor drops slightly while executing these instructions but not so much as to make it slower than not having the instruction. New instruction sets usually do two things - add support for new operations, and widen the registers and allow you to perform the operation on more elements at a time. If you’re writing in a high level language, you’ll only really see performance improvements if your interpreter or library takes advantage of them. For e.g. the Python library NumPy makes good use of vectorisation. In terms of writing low level code, simple loops are often auto-vectorised by the compiler so you often see a sort of odd style where a loop doing many things is split up so the compiler can deal with it. You often end up having to run the code through something like VTune to work out whether a particular loop has actually vectorised.
crote 3y ago
Anything which works on nontrivial datasets, really. As soon as you need to do roughly the same operation on a bunch of data, you can benefit from AVX. Heck, this could be as simply as determining the length of a string! The main benefit of AVX-512 is that the CPU gained support for "masking". Traditionally, you had to execute the instructions on all data elements. Something like "round all even values downwards, round all odd values upwards" becomes near-impossible, and if you want to do that in an entire AVX pipeline you have to deconstruct the vector into individual elements, do the operation on each of them individually, and reconstruct it into a vector. This really sucks. With masking, you get a special field which specify on which elements the operation should apply. So you could just create a mask with the even values, use that mask for a round-down, invert the mask, and use it for a round-up. That's a significant speedup! Adding AVX-512 support means that a lot more applications suddenly become eligible for fairly trivial vectorization, and I personally can't wait for it to become universally available.