3 ms·

Like another poster already said, the power budget of a consumer CPU, like 9950X, executing programs at a double clock frequency in comparison with a GPU, allow

Like another poster already said, the power budget of a consumer CPU, like 9950X, executing programs at a double clock frequency in comparison with a GPU, allows for 16 cores x 2 execution units x 16 = 512 FP32 FMA per clock cycle, which provides the same throughput like an 1024 FP32 FMA per clock cycle iGPU from the best laptop CPUs, while consuming 3 times less power than a datacenter GPU, so the power budget and performance is like for a datacenter GPU with 3072 FP32 FMA per clock cycle. However, because of its high clock frequency a consumer CPU has high performance per dollar, but low performance per watt. Server CPUs with many cores have much better energy efficiency, e.g. around 3 times higher than a desktop CPU and the same with the most efficient laptop CPUs. For many generations of NVIDIA GPUs and Intel Xeon CPUs, until about 5-6 years ago, the ratio between their floating-point FMA throughput per watt has been of only 3. This factor of 3 is mainly due to the overhead of various tricks used by CPUs to extract instruction-level parallelism from programs that do not use enough concurrent threads or array operations, e.g. superscalar out-of-order execution, register renaming, etc. In recent years, starting with NVIDIA Volta, followed later by AMD and Intel GPUs, the GPUs have made a jump in performance that has increased the gap between their throughput and that of CPUs, by supplementing the vector instructions with matrix instructions, i.e. what NVIDIA calls tensor instructions. However this current greater gap in performance between CPUs and GPUs could easily be removed and the performance per watt ratio could be brought back to a factor unlikely to be greater than 3, by adding matrix instructions to the CPUs. Intel has introduced the AMX instruction set, besides AVX, but for now it is supported only in expensive server CPUs and Intel has defined only instructions for low-precision operations used for AI/ML. If AMX were extended with FP32 and FP64 operations, then the performance would be much more competitive with GPUs. ARM is more advanced in this direction, with SME (Scalable Matrix Extension) defined besides SVE (Scalable Vector Extension). SME is already available in recent Apple CPUs and it is expected to be also available in the new Arm cores that will be announced in a few months for now, which should become available in the smartphones of 2026, and presumably also in future Arm-based CPUs for servers and laptops. The current Apple CPUs do not have strong SME accelerators, because they also have an iGPU that can perform the operations whose latency is less important. On the other hand, an Arm-based server CPU could have a much bigger SME accelerator, providing a performance much closer to a GPU.

menaerus 2y ago
I appreciate the response with a lot of interesting details, however, I don't believe it answers the question I had? My doubt was why is it so that the CPU design suffers from clock frequency issues in AVX-512 workloads whereas GPUs which have much more compute power do not. I assumed that it was due to the fact that GPUs run at much lower clock frequencies and therefore available power budget but as I also discussed with another commenter above this was probably a premature conclusion since we don't have enough evidence showing that GPUs indeed do not suffer from same type of issues. They likely do but nobody measured it yet?
- adrian_b 2y ago
  The low clock frequency when executing AVX-512 workloads is a frequency where the CPU operates efficiently, with a low energy consumption per operation executed. For such a workload that executes a very large number of operations per second, the CPU cannot afford to operate inefficiently because it will overheat. When a CPU core has many execution units that are idle, so they do not consume power, like when executing only scalar operations or only operations with narrow 128-bit vectors, it can afford to raise the clock frequency e.g. by 50%, even if that would increase the energy consumption per operation e.g. 3 times. By executing 4 times or 8 times less operations per clock cycle, even if the energy consumption is 3 times higher the total power consumption is smaller and the CPU does not overheat and the desktop owner does not care that the completion of the same workload requires much more energy, because it is likely that the owner cares more about the time to completion. The clock frequency of a GPU also varies continuously depending on the workload, in order to maintain the power consumption within the limits. However a GPU is not designed to be able to increase the clock frequency as much as a CPU. The fastest GPUs have clock frequencies under 3 GHz, while the fastest CPUs exceed 6 GHz. The reason is that normally one never launches a GPU program that would use only a small fraction of the resources of a GPU allowing a higher clock frequency, so it makes no sense to design a GPU for this use case. Designing a chip for a higher clock frequency greatly increases the size of the chip, as shown by the comparison between a normal Zen core designed for 5.7 GHz and a Zen compact core, designed e.g. for 3.3 GHz, a frequency not much higher than that of a GPU. On Zen compact cores and on normal Zen cores configured for server CPUs with a large number of cores, e.g. 128 cores (with a total of 4096 FP32 ALUs, like a low-to-mid-range desktop GPU, or like a top desktop GPU of 5 years ago; a Zen compact server CPU can have 6144 FP32 ALUs, more than a RTX 4070), the clock frequency variation range is small, very similar to the clock variation range of a GPU. In conclusion, it is not the desktop/laptop CPUs which drop their clock frequency, but it is the GPUs which never raise their clock frequency much, the same as the server CPUs, because neither GPUs nor server CPUs are normally running programs that keep most of their execution units idle, to allow higher clock frequencies without overheating.

menaerus 2y ago

I appreciate the response with a lot of interesting details, however, I don't believe it answers the question I had? My doubt was why is it so that the CPU design suffers from clock frequency issues in AVX-512 workloads whereas GPUs which have much more compute power do not. I assumed that it was due to the fact that GPUs run at much lower clock frequencies and therefore available power budget but as I also discussed with another commenter above this was probably a premature conclusion since we don't have enough evidence showing that GPUs indeed do not suffer from same type of issues. They likely do but nobody measured it yet?

adrian_b 2y ago
The low clock frequency when executing AVX-512 workloads is a frequency where the CPU operates efficiently, with a low energy consumption per operation executed. For such a workload that executes a very large number of operations per second, the CPU cannot afford to operate inefficiently because it will overheat. When a CPU core has many execution units that are idle, so they do not consume power, like when executing only scalar operations or only operations with narrow 128-bit vectors, it can afford to raise the clock frequency e.g. by 50%, even if that would increase the energy consumption per operation e.g. 3 times. By executing 4 times or 8 times less operations per clock cycle, even if the energy consumption is 3 times higher the total power consumption is smaller and the CPU does not overheat and the desktop owner does not care that the completion of the same workload requires much more energy, because it is likely that the owner cares more about the time to completion. The clock frequency of a GPU also varies continuously depending on the workload, in order to maintain the power consumption within the limits. However a GPU is not designed to be able to increase the clock frequency as much as a CPU. The fastest GPUs have clock frequencies under 3 GHz, while the fastest CPUs exceed 6 GHz. The reason is that normally one never launches a GPU program that would use only a small fraction of the resources of a GPU allowing a higher clock frequency, so it makes no sense to design a GPU for this use case. Designing a chip for a higher clock frequency greatly increases the size of the chip, as shown by the comparison between a normal Zen core designed for 5.7 GHz and a Zen compact core, designed e.g. for 3.3 GHz, a frequency not much higher than that of a GPU. On Zen compact cores and on normal Zen cores configured for server CPUs with a large number of cores, e.g. 128 cores (with a total of 4096 FP32 ALUs, like a low-to-mid-range desktop GPU, or like a top desktop GPU of 5 years ago; a Zen compact server CPU can have 6144 FP32 ALUs, more than a RTX 4070), the clock frequency variation range is small, very similar to the clock variation range of a GPU. In conclusion, it is not the desktop/laptop CPUs which drop their clock frequency, but it is the GPUs which never raise their clock frequency much, the same as the server CPUs, because neither GPUs nor server CPUs are normally running programs that keep most of their execution units idle, to allow higher clock frequencies without overheating.