3 ms·

I don't understand why 2x FMAs in CPU design poses such a challenge when GPUs literally have hundreds of such ALUs? Both operate at similar TDP so where's the c

I don't understand why 2x FMAs in CPU design poses such a challenge when GPUs literally have hundreds of such ALUs? Both operate at similar TDP so where's the catch? Much lower GPU clock frequency?

eqvinox 2y ago
It's not 2 FMAs, it's AVX-512 (and going with 32-bit words) ⇒ 2*512/32 = 32 FMAs per core, 256 on an 8-core CPU. The unit counts for GPUs - depending on which number you look at - count these separately. CPUs also have much more complicated program flow control, versatility, and AFAIK latency (⇒ flow control cost) of individual instructions. GPUs are optimized for raw calculation throughput meanwhile. Also note that modern GPUs and CPUs don't have a clear pricing relationship anymore, e.g. a desktop CPU is much cheaper than a high-end GPU, and large server CPUs are more expensive than either.
- menaerus 2y ago
  1x 512-bit FMA or 2x 256-bit FMAs or 4x 128-bit FMAs is irrelevant here - it's still a single physical unit in a CPU that consumes 512 bits of data bandwidth. The question is why the CPU budget allows for 2x 512-bit or 4x 256-bit while H100, for example, has 14592 FP32 CUDA cores - in AVX terminology that would translate, if I am not mistaken, to 7926x 512-bit or 14592x 256-bit FMAs per clock cycle. Even considering the obvious differences between GPUs and CPUs, this is still a large difference. Since GPU cores operate at much lower frequencies than CPU cores, it is what it made me believe where the biggest difference comes from.
  - eqvinox 2y ago
    AIUI an FP32 core is only 32 bits wide, but this is outside my area of expertise really. Also note that CPUs also have additional ALUs that can't do FMAs, FMA is just the most capable one. You're also repeating 2×512 / 4×256 — that's per core, you need to multiply by CPU core count. [also, note e.g. an 8-core CPU is much cheaper than a H100 card ;) — if anything you'd be comparing the highest end server CPUs here. An 192-core Zen5c is 8.2~10.5k€ open retail, an H100 is 32~35k€…] [reading through some random docs, a CPU core seems vaguely comparable to a SM; a SM might have 128 or 64 lanes (=FP32 cores) while a CPU only has 16 with AVX-512, but there is indeed also a notable clock difference and far more flexibility otherwise in the CPU core (which consumes silicon area)]
    - deleted 2y ago
      [deleted]
  - adrian_b 2y ago
    Like another poster already said, the power budget of a consumer CPU, like 9950X, executing programs at a double clock frequency in comparison with a GPU, allows for 16 cores x 2 execution units x 16 = 512 FP32 FMA per clock cycle, which provides the same throughput like an 1024 FP32 FMA per clock cycle iGPU from the best laptop CPUs, while consuming 3 times less power than a datacenter GPU, so the power budget and performance is like for a datacenter GPU with 3072 FP32 FMA per clock cycle. However, because of its high clock frequency a consumer CPU has high performance per dollar, but low performance per watt. Server CPUs with many cores have much better energy efficiency, e.g. around 3 times higher than a desktop CPU and the same with the most efficient laptop CPUs. For many generations of NVIDIA GPUs and Intel Xeon CPUs, until about 5-6 years ago, the ratio between their floating-point FMA throughput per watt has been of only 3. This factor of 3 is mainly due to the overhead of various tricks used by CPUs to extract instruction-level parallelism from programs that do not use enough concurrent threads or array operations, e.g. superscalar out-of-order execution, register renaming, etc. In recent years, starting with NVIDIA Volta, followed later by AMD and Intel GPUs, the GPUs have made a jump in performance that has increased the gap between their throughput and that of CPUs, by supplementing the vector instructions with matrix instructions, i.e. what NVIDIA calls tensor instructions. However this current greater gap in performance between CPUs and GPUs could easily be removed and the performance per watt ratio could be brought back to a factor unlikely to be greater than 3, by adding matrix instructions to the CPUs. Intel has introduced the AMX instruction set, besides AVX, but for now it is supported only in expensive server CPUs and Intel has defined only instructions for low-precision operations used for AI/ML. If AMX were extended with FP32 and FP64 operations, then the performance would be much more competitive with GPUs. ARM is more advanced in this direction, with SME (Scalable Matrix Extension) defined besides SVE (Scalable Vector Extension). SME is already available in recent Apple CPUs and it is expected to be also available in the new Arm cores that will be announced in a few months for now, which should become available in the smartphones of 2026, and presumably also in future Arm-based CPUs for servers and laptops. The current Apple CPUs do not have strong SME accelerators, because they also have an iGPU that can perform the operations whose latency is less important. On the other hand, an Arm-based server CPU could have a much bigger SME accelerator, providing a performance much closer to a GPU.
    - 2y ago
  - TinkersW 2y ago
    Nvidia calls them cores to deliberately confuse people, and make it appear vastly more powerful than it really is. What they are in reality is SIMD lanes. So the H100(which costs vastly more than a Zen5..), has 14592 32 bit SIMD lanes, not cores. A Zen 5 has 16x4(64) 32 bit SIMD lanes per core, so scale that by core count to get your answer. A higher end desktop Zen5 will have 16 cores, so 64x16 = 1024. The Zen5 also clocks much higher than the GPU, so you can also scale it up by perhaps 1.5-2x While this is obviously less than the H100, the Zen5 chip costs $550 and the H100 cost $40k. There is more to it than this, GPUs also have transcendental functions, texture sampling, and 16 bit ops(which are lacking in CPUs). While CPUs are much more flexible, and have powerful byte & integer manipulation instructions, along with full speed 64 bit integer/double support.
    - dzaima 2y ago
      There are x86 extensions for fp16/bf16 ops. e.g. both Zen 4 and Zen 5 support AVX512_BF16, which has vdpbf16ps, i.e. dot product of pairs of bf16 elements from two args; that is, takes a total of 64 bf16 elts and outputs 16 fp32 elts. Zen 5 can run two such instrs per cycle.
    - menaerus 2y ago
      Thanks for the clarification on the NVidia, I didn't know that. What I also found is that NVidia groups 32 SIMD lanes into what they call a warp. Then 4 warps are grouped into what they're calling a streaming multiprocessor (SM). And lastly H100 has 114 SMs so 432114=14592 checks out. > Zen 5 has 16x4(64) 32 bit SIMD lanes per core Right, 2x FMA and 2x FADD so the highest-end Zen 5 die with 192 cores would total to 12288 32-bit SIMD lanes or half of that if we are considering only FMA ops. This is then indeed much closer to 14592 32-bit SIMD lanes of H100.
      - deleted 2y ago
        [deleted]
atq2119 2y ago
It's not the execution of FMAs that's the challenge, it's the ramp up / down. And I assure you GPUs do have challenges with that as well. That's just less well known because (1) in GPUs, all workloads are vector workloads, and so there was never a stark contrast between scalar and vector regimes like in Intel's AVX-512 implementation and (2) GPU performance characteristics are in general less broadly known.
- menaerus 2y ago
  Yes, I agree that it was premature to say that GPUs aren't suffering from the same symptoms. There's just not enough evidence but the differences in the compute power are still large.
  - TinkersW 2y ago
    If you normalize for cost, the difference isn't nearly as large as you might think, around 4-5x.
dzaima 2y ago
Zen 5 still clocks way higher than GPUs even with the penalties. Additionally, CPUs typically target much lower latency for operations even per-clock, which adds a ton of silicon cost for the same throughput, and especially so at high clock frequency. The difficulty with transitions that Skylake-X suffered especially from just has no equivalent on GPU; if you always stay in the transitioned-to-AVX512 state on Skylake-X, things are largely normal; GPUs just are always unconditionally in such a state, but that be awful on CPUs, as it'd make scalar-only code (not a thing on GPUs, but the main target for CPUs) unnecessarily slow. And so Intel decided that the transitions are worth the improved clocks for code not utilizing AVX-512.
adgjlsfhk1 2y ago
It's gpu frequency 5.5 GHZ is ~4x the heat and power as the 2.5 GHZ for GPUs.

I don't understand why 2x FMAs in CPU design poses such a challenge when GPUs literally have hundreds of such ALUs? Both operate at similar TDP so where's the catch? Much lower GPU clock frequency?

eqvinox 2y ago

It's not 2 FMAs, it's AVX-512 (and going with 32-bit words) ⇒ 2*512/32 = 32 FMAs per core, 256 on an 8-core CPU. The unit counts for GPUs - depending on which number you look at - count these separately. CPUs also have much more complicated program flow control, versatility, and AFAIK latency (⇒ flow control cost) of individual instructions. GPUs are optimized for raw calculation throughput meanwhile. Also note that modern GPUs and CPUs don't have a clear pricing relationship anymore, e.g. a desktop CPU is much cheaper than a high-end GPU, and large server CPUs are more expensive than either.

menaerus 2y ago
1x 512-bit FMA or 2x 256-bit FMAs or 4x 128-bit FMAs is irrelevant here - it's still a single physical unit in a CPU that consumes 512 bits of data bandwidth. The question is why the CPU budget allows for 2x 512-bit or 4x 256-bit while H100, for example, has 14592 FP32 CUDA cores - in AVX terminology that would translate, if I am not mistaken, to 7926x 512-bit or 14592x 256-bit FMAs per clock cycle. Even considering the obvious differences between GPUs and CPUs, this is still a large difference. Since GPU cores operate at much lower frequencies than CPU cores, it is what it made me believe where the biggest difference comes from.
- eqvinox 2y ago
  AIUI an FP32 core is only 32 bits wide, but this is outside my area of expertise really. Also note that CPUs also have additional ALUs that can't do FMAs, FMA is just the most capable one. You're also repeating 2×512 / 4×256 — that's per core, you need to multiply by CPU core count. [also, note e.g. an 8-core CPU is much cheaper than a H100 card ;) — if anything you'd be comparing the highest end server CPUs here. An 192-core Zen5c is 8.2~10.5k€ open retail, an H100 is 32~35k€…] [reading through some random docs, a CPU core seems vaguely comparable to a SM; a SM might have 128 or 64 lanes (=FP32 cores) while a CPU only has 16 with AVX-512, but there is indeed also a notable clock difference and far more flexibility otherwise in the CPU core (which consumes silicon area)]
  - deleted 2y ago
    [deleted]
- adrian_b 2y ago
  Like another poster already said, the power budget of a consumer CPU, like 9950X, executing programs at a double clock frequency in comparison with a GPU, allows for 16 cores x 2 execution units x 16 = 512 FP32 FMA per clock cycle, which provides the same throughput like an 1024 FP32 FMA per clock cycle iGPU from the best laptop CPUs, while consuming 3 times less power than a datacenter GPU, so the power budget and performance is like for a datacenter GPU with 3072 FP32 FMA per clock cycle. However, because of its high clock frequency a consumer CPU has high performance per dollar, but low performance per watt. Server CPUs with many cores have much better energy efficiency, e.g. around 3 times higher than a desktop CPU and the same with the most efficient laptop CPUs. For many generations of NVIDIA GPUs and Intel Xeon CPUs, until about 5-6 years ago, the ratio between their floating-point FMA throughput per watt has been of only 3. This factor of 3 is mainly due to the overhead of various tricks used by CPUs to extract instruction-level parallelism from programs that do not use enough concurrent threads or array operations, e.g. superscalar out-of-order execution, register renaming, etc. In recent years, starting with NVIDIA Volta, followed later by AMD and Intel GPUs, the GPUs have made a jump in performance that has increased the gap between their throughput and that of CPUs, by supplementing the vector instructions with matrix instructions, i.e. what NVIDIA calls tensor instructions. However this current greater gap in performance between CPUs and GPUs could easily be removed and the performance per watt ratio could be brought back to a factor unlikely to be greater than 3, by adding matrix instructions to the CPUs. Intel has introduced the AMX instruction set, besides AVX, but for now it is supported only in expensive server CPUs and Intel has defined only instructions for low-precision operations used for AI/ML. If AMX were extended with FP32 and FP64 operations, then the performance would be much more competitive with GPUs. ARM is more advanced in this direction, with SME (Scalable Matrix Extension) defined besides SVE (Scalable Vector Extension). SME is already available in recent Apple CPUs and it is expected to be also available in the new Arm cores that will be announced in a few months for now, which should become available in the smartphones of 2026, and presumably also in future Arm-based CPUs for servers and laptops. The current Apple CPUs do not have strong SME accelerators, because they also have an iGPU that can perform the operations whose latency is less important. On the other hand, an Arm-based server CPU could have a much bigger SME accelerator, providing a performance much closer to a GPU.
  - 2y ago
- TinkersW 2y ago
  Nvidia calls them cores to deliberately confuse people, and make it appear vastly more powerful than it really is. What they are in reality is SIMD lanes. So the H100(which costs vastly more than a Zen5..), has 14592 32 bit SIMD lanes, not cores. A Zen 5 has 16x4(64) 32 bit SIMD lanes per core, so scale that by core count to get your answer. A higher end desktop Zen5 will have 16 cores, so 64x16 = 1024. The Zen5 also clocks much higher than the GPU, so you can also scale it up by perhaps 1.5-2x While this is obviously less than the H100, the Zen5 chip costs $550 and the H100 cost $40k. There is more to it than this, GPUs also have transcendental functions, texture sampling, and 16 bit ops(which are lacking in CPUs). While CPUs are much more flexible, and have powerful byte & integer manipulation instructions, along with full speed 64 bit integer/double support.
  - dzaima 2y ago
    There are x86 extensions for fp16/bf16 ops. e.g. both Zen 4 and Zen 5 support AVX512_BF16, which has vdpbf16ps, i.e. dot product of pairs of bf16 elements from two args; that is, takes a total of 64 bf16 elts and outputs 16 fp32 elts. Zen 5 can run two such instrs per cycle.
  - menaerus 2y ago
    Thanks for the clarification on the NVidia, I didn't know that. What I also found is that NVidia groups 32 SIMD lanes into what they call a warp. Then 4 warps are grouped into what they're calling a streaming multiprocessor (SM). And lastly H100 has 114 SMs so 432114=14592 checks out. > Zen 5 has 16x4(64) 32 bit SIMD lanes per core Right, 2x FMA and 2x FADD so the highest-end Zen 5 die with 192 cores would total to 12288 32-bit SIMD lanes or half of that if we are considering only FMA ops. This is then indeed much closer to 14592 32-bit SIMD lanes of H100.
    - deleted 2y ago
      [deleted]

atq2119 2y ago

It's not the execution of FMAs that's the challenge, it's the ramp up / down. And I assure you GPUs do have challenges with that as well. That's just less well known because (1) in GPUs, all workloads are vector workloads, and so there was never a stark contrast between scalar and vector regimes like in Intel's AVX-512 implementation and (2) GPU performance characteristics are in general less broadly known.

menaerus 2y ago
Yes, I agree that it was premature to say that GPUs aren't suffering from the same symptoms. There's just not enough evidence but the differences in the compute power are still large.
- TinkersW 2y ago
  If you normalize for cost, the difference isn't nearly as large as you might think, around 4-5x.

dzaima 2y ago

Zen 5 still clocks way higher than GPUs even with the penalties. Additionally, CPUs typically target much lower latency for operations even per-clock, which adds a ton of silicon cost for the same throughput, and especially so at high clock frequency. The difficulty with transitions that Skylake-X suffered especially from just has no equivalent on GPU; if you always stay in the transitioned-to-AVX512 state on Skylake-X, things are largely normal; GPUs just are always unconditionally in such a state, but that be awful on CPUs, as it'd make scalar-only code (not a thing on GPUs, but the main target for CPUs) unnecessarily slow. And so Intel decided that the transitions are worth the improved clocks for code not utilizing AVX-512.

adgjlsfhk1 2y ago

It's gpu frequency 5.5 GHZ is ~4x the heat and power as the 2.5 GHZ for GPUs.