3 ms·

Great in-depth article. One especially interesting data point relevant to ongoing discussions about AVX-512 area cost: 40% of 2.93 mm2 per core is AVX-512, so

by janwas 4y ago

Great in-depth article. One especially interesting data point relevant to ongoing discussions about AVX-512 area cost: 40% of 2.93 mm2 per core is AVX-512, so 1.14 mm2. This is a large fraction, but as the article says the core is basically a minimum wrapper around the vector unit, with rather weak L1i/branch predictor/store buffers. Let's put that in the context of modern chips. 14nm density was 44.67 MTr/mm2 so that's 50.9 MTr for AVX-512. To compare with 5nm, let's use density of TSMC N5 (138.2 MTr/mm2) to get 0.37 mm2. So that's about 10% of an Apple M1 Firestorm core to enable 5-10x speedups vs scalar code. Sounds worthwhile to me. We can now stop saying that "AVX-512 is a huge fraction of modern cores", and "give us more cores instead". Let's instead use the hardware we have :)

rbanffy 4y ago
> "AVX-512 is a huge fraction of modern cores", and "give us more cores instead". Let's instead use the hardware we have If what you do can use a wide SIMD pipeline, then, by all means, get AVX-512. If what you want is to have lots of branchy processes running serving different things, then more cores is a better idea. I'd suggest the best of both worlds - asymmetric cores. Put a couple that have wide SIMD pipelines, alongside a couple others that don't, but may be smaller and more numerous. AVX-512 tasks that are scheduled to non-AVX-512 trap and are scheduled to AVX-512 cores. Tasks that don't trap continue running wherever the kernel thinks is best.
ksec 4y ago
Yes. That was part of the plan for Intel. Remember that Intel was suppose to have 7nm by 2019, which is roughly equal to TSMC N5.
celrod 4y ago
Good analysis. It's also worth pointing out that this is for 2x 512 bit FMA, which is more than client Ice/Tiger/Rocket lake or Zen4 have. Personally, I bought HEDT (Skylake-X and Cascadelake) because I wanted 2x 512 bit AVX512. Glad it's cheap in terms of area, and I'm hoping we'll get more options with great vector performance in the future.
- janwas 4y ago
  Good point about the second FMA. I'm not certain it's the best tradeoff, Genoa only has two half-width FMA. I share your hope for more focus on vectors. It's also up to us software devs, CPUs will not invest as heavily if we don't use it.
  - celrod 4y ago
    I do think that Genoa's approach is a reasonable one. I'd like to see one of Gracemont's successors doing the same. Maybe we'll even see quadruple pumping for AVX-512 some day? I'll be impressed if/when an Atom line CPU gets 4x 128bit fma units to match ARM's Cortex-X line or Apple's Firestorm). I think these are good options, and can allow AVX-512 to sort of act like SVE, but with the benefits of a fixed size architecture (i.e., shuffles); compile one set of code and you're able to run it anywhere, with performance dictated by how much the vendor decided was worth investing into the vector units. And AVX512 can still help (like it does Genoa) by taking a lot of pressure off of the front end. I'm also trying to do my part as a software dev! I wrote/maintain the JuliaSIMD ecosystem, and working on good loop vectorizers to let people take advantage of their vector units is my passion; LoopVectorization.jl has gotten great results on many benchmarks[0], and I'm rewriting it as an LLVM pass to try and address as many of its flaws and limitations as I can. [0] For example, in a simple self-dot-product benchmark, LLVM's 256 bit code is actually faster than its 512 bit code when testing random sizes from 1-256: https://github.com/JuliaSIMD/LoopVectorization.jl/issues/446#issuecomment-1331500465 https://github.com/JuliaSIMD/LoopVectorization.jl/issues/446... However, LoopVectorization.jl's 512 bit code is close to twice as fast as either LLVM's 256 bit or 512 bit code. This is a trivial example; the difference can be much larger for more complicated code.
    - janwas 4y ago
      > Maybe we'll even see quadruple pumping for AVX-512 some day? > can allow AVX-512 to sort of act like SVE, but with the benefits of a fixed size architecture (i.e., shuffles); compile one set of code and you're able to run it anywhere, with performance dictated by how much the vendor decided was worth investing into the vector units That makes a lot of sense. It's basically the equivalent of RISC-V's LMUL=4, with the big advantage of reducing instruction count as you say. That seems a better route than 4x128, which might actually be less in practice if there are resource conflicts. > I'm also trying to do my part as a software dev! I wrote/maintain the JuliaSIMD ecosystem That's awesome, congrats on the good result. Looks like your preference is to allow people to write high-level code without much worry about the arch details. Any thoughts on how we can spread awareness of the basics such as data-oriented programming (avoiding branches, optimizing for cache and contiguous memory accesses)?
moonchild 4y ago
> give us more cores instead That should be a pretty obvious conclusion--higher-clocked cores is better than wider cores, and wider cores is better than more cores. Communication cost between concurrency domains for superscalar/supervector: min <1ns Communication cost between concurrency domains for multiprocessors: min 10-100ns
- janwas 4y ago
  :) Obvious or not, one does still occasionally hear "give us more cores instead". I agree about the communication cost. Higher clocks are harder - quadratic increase in power. For shared-nothing problems, an array of wimpy (low frequency) cores is pretty good. It seems to me that wider, lower-frequency cores are a good compromise: beefy enough (thanks to vectors) to reduce the number of cores required, while still power-efficient due to both vectors and lower clocks.
phonon 4y ago
Knight’s Landing AVX-512 units run at about 1.5 GHz at peak (even when only a few are in use.) The ones in Sapphire Rapids run 2x+ faster. Likely there is more pipelining and many more transistors used to reach those clock speeds.
- janwas 4y ago
  Physical design is outside my area of expertise, but I understand that a large part of the increase may be due to improved fab. 15-20% speed improvement per node would account for the majority of the clock speed gain. Also, with smaller feature sizes comes the ability to trade away some of the density for further increases in the frequency, without actually requiring more transistors.

by janwas 4y ago

rbanffy 4y ago

> "AVX-512 is a huge fraction of modern cores", and "give us more cores instead". Let's instead use the hardware we have If what you do can use a wide SIMD pipeline, then, by all means, get AVX-512. If what you want is to have lots of branchy processes running serving different things, then more cores is a better idea. I'd suggest the best of both worlds - asymmetric cores. Put a couple that have wide SIMD pipelines, alongside a couple others that don't, but may be smaller and more numerous. AVX-512 tasks that are scheduled to non-AVX-512 trap and are scheduled to AVX-512 cores. Tasks that don't trap continue running wherever the kernel thinks is best.

ksec 4y ago

Yes. That was part of the plan for Intel. Remember that Intel was suppose to have 7nm by 2019, which is roughly equal to TSMC N5.

celrod 4y ago

Good analysis. It's also worth pointing out that this is for 2x 512 bit FMA, which is more than client Ice/Tiger/Rocket lake or Zen4 have. Personally, I bought HEDT (Skylake-X and Cascadelake) because I wanted 2x 512 bit AVX512. Glad it's cheap in terms of area, and I'm hoping we'll get more options with great vector performance in the future.

janwas 4y ago
Good point about the second FMA. I'm not certain it's the best tradeoff, Genoa only has two half-width FMA. I share your hope for more focus on vectors. It's also up to us software devs, CPUs will not invest as heavily if we don't use it.
- celrod 4y ago
  I do think that Genoa's approach is a reasonable one. I'd like to see one of Gracemont's successors doing the same. Maybe we'll even see quadruple pumping for AVX-512 some day? I'll be impressed if/when an Atom line CPU gets 4x 128bit fma units to match ARM's Cortex-X line or Apple's Firestorm). I think these are good options, and can allow AVX-512 to sort of act like SVE, but with the benefits of a fixed size architecture (i.e., shuffles); compile one set of code and you're able to run it anywhere, with performance dictated by how much the vendor decided was worth investing into the vector units. And AVX512 can still help (like it does Genoa) by taking a lot of pressure off of the front end. I'm also trying to do my part as a software dev! I wrote/maintain the JuliaSIMD ecosystem, and working on good loop vectorizers to let people take advantage of their vector units is my passion; LoopVectorization.jl has gotten great results on many benchmarks[0], and I'm rewriting it as an LLVM pass to try and address as many of its flaws and limitations as I can. [0] For example, in a simple self-dot-product benchmark, LLVM's 256 bit code is actually faster than its 512 bit code when testing random sizes from 1-256: https://github.com/JuliaSIMD/LoopVectorization.jl/issues/446#issuecomment-1331500465 https://github.com/JuliaSIMD/LoopVectorization.jl/issues/446... However, LoopVectorization.jl's 512 bit code is close to twice as fast as either LLVM's 256 bit or 512 bit code. This is a trivial example; the difference can be much larger for more complicated code.
  - janwas 4y ago
    > Maybe we'll even see quadruple pumping for AVX-512 some day? > can allow AVX-512 to sort of act like SVE, but with the benefits of a fixed size architecture (i.e., shuffles); compile one set of code and you're able to run it anywhere, with performance dictated by how much the vendor decided was worth investing into the vector units That makes a lot of sense. It's basically the equivalent of RISC-V's LMUL=4, with the big advantage of reducing instruction count as you say. That seems a better route than 4x128, which might actually be less in practice if there are resource conflicts. > I'm also trying to do my part as a software dev! I wrote/maintain the JuliaSIMD ecosystem That's awesome, congrats on the good result. Looks like your preference is to allow people to write high-level code without much worry about the arch details. Any thoughts on how we can spread awareness of the basics such as data-oriented programming (avoiding branches, optimizing for cache and contiguous memory accesses)?

moonchild 4y ago

> give us more cores instead That should be a pretty obvious conclusion--higher-clocked cores is better than wider cores, and wider cores is better than more cores. Communication cost between concurrency domains for superscalar/supervector: min <1ns Communication cost between concurrency domains for multiprocessors: min 10-100ns

janwas 4y ago
:) Obvious or not, one does still occasionally hear "give us more cores instead". I agree about the communication cost. Higher clocks are harder - quadratic increase in power. For shared-nothing problems, an array of wimpy (low frequency) cores is pretty good. It seems to me that wider, lower-frequency cores are a good compromise: beefy enough (thanks to vectors) to reduce the number of cores required, while still power-efficient due to both vectors and lower clocks.

phonon 4y ago

Knight’s Landing AVX-512 units run at about 1.5 GHz at peak (even when only a few are in use.) The ones in Sapphire Rapids run 2x+ faster. Likely there is more pipelining and many more transistors used to reach those clock speeds.

janwas 4y ago
Physical design is outside my area of expertise, but I understand that a large part of the increase may be due to improved fab. 15-20% speed improvement per node would account for the majority of the clock speed gain. Also, with smaller feature sizes comes the ability to trade away some of the density for further increases in the frequency, without actually requiring more transistors.