3 ms·

If SIMD lanes come to vastly dominate the composition of a typical computer chip (in terms, e.g., of where power is consumed) will the distinction between CPU/G

If SIMD lanes come to vastly dominate the composition of a typical computer chip (in terms, e.g., of where power is consumed) will the distinction between CPU/GPU continue to be meaningful? For decades the GPU was "special purpose" hardware dedicated to the math of screen graphics. If the type of larger-scale numerical computation that has been popularised with LLM's is now deemed "typical use", then the distinction may be becoming irrelevant (and even counterproductive from a software development perspective).

JonChesterfield 2y ago
The x64 cores putting more hardware into the vector units and amdgpu changing from 64 wide to 32 wide simd (at least for some chips) looks like convergent evolution to me. My personal belief is that the speculation and pipelining approach is worse than the many tasks and swapping between them. I think the APU designs from AMD are the transition pointing to the future. The GPU cores will gain increasing access to the raw hardware and the user interface until the CPU cores are optional and ultimately discarded.
- adrian_b 2y ago
  There is little relationship between the reasons that determine the width of SIMD in CPUs and GPUs, so there is no convergence between them. In the Intel/AMD CPUs, the 512-bit width, i.e. 64 bytes or 16 FP32 numbers, matches the width of the cache line and the width of a DRAM burst transfer, which simplifies the writing of optimized programs. This SIMD width also provides a good ratio between the power consumed in the execution units and the power wasted in the control part of the CPU (around 80% of the total power consumption goes to the execution units, which is much more than when using narrower SIMD instructions). Increasing the SIMD width more than that in CPUs would complicate the interaction with the cache memories and with the main memory, while providing only a negligible improvement in the energy efficiency, so there is no reason to do this. At least in the following decade it is very unlikely that any CPU would increase the SIMD width beyond 16 FP32 numbers per operation. On the other hand, the AMD GPUs before RDNA had a SIMD width of 64 FP32 numbers, but the operations were pipelined and executed in 4 clock cycles, so only 16 FP32 numbers were processed per clock cycle. RDNA has doubled the width of the SIMD execution, processing 32 FP32 numbers per clock cycle. For this, SIMD instructions with a reduced width of 32 FP32 have been introduced, but they are executed in one clock cycle versus the old 64 FP32 instructions that were executed in four clock cycles. For backwards compatibility, RDNA has kept 64 FP32 instructions, which are executed in two clock cycles, but these were not recommended for new programs. RDNA 3 has changed again all this, because now sometimes the 64 FP32 instructions can be executed in a single clock cycle, so they may be again preferable instead of the 32 FP32 instructions. However it is possible to take advantage of the increased width of the RDNA 3 SIMD execution units also when using 32 FP32 instructions, if certain new instructions are used, which encode double operations. So the AMD GPUs have continuously evolved towards wider SIMD execution units, from 16 FP32 before RDNA, to 32 FP32 in RDNA and finally to 64 FP32 in RDNA 3. The distance from CPUs has been steadily increasing, there is no convergence.
Symmetry 2y ago
There are still a lot of differences, even if you put in a lot more SIMD lanes to the CPU. CPUs keep their execution resources fed by by aggressive caching, prefetching, and out of order execution while GPUs rely on having lots of threads around so that if one stalls another is able to execute.
bee_rider 2y ago
Xeon Phi was that, a bunch of little cores with a ton of SIMD lanes each. It didn’t really work out, in part because it was too far from a regular old Xeon to run your normal code well without optimizing. On the other side, Intel couldn’t keep up with NVIDIA on the sort of metrics people care about for these compute accelerators: memory bandwidth mostly. If you are going to have to refactor your whole project anyway to use a compute accelerator, you probably want a pretty big reward. It isn’t obvious (to me at least) if this is the result of the fact that the Phi cores, simple as they were, were still a lot more complex than a GPU “core,” maybe the design just had various hidden bottlenecks that were too hard to work out due to that complexity. Or if it is because Intel just wasn’t executing very well at the time, especially compared to NVIDIA (it is Intel’s dark age vs NVIDIA’s golden age, really). The programmer’s “logical or” joke is a possible answer here. But, you can’t do everything in parallel. It is a shame the Phi didn’t survive into the age where Intel is also doing big/little cores (in a single chip). A big Xeon core (for latency) surrounded by a bunch of little Phi cores (for throughput) could have been a really interesting device.
dahart 2y ago
The special purpose graphics distinction is already mostly irrelevant and has been for 10 or 20 years for anyone doing High Performance Computing (HPC) or AI. It predates LLMs. For a while we had the acronym GPGPU - General Purpose computing on Graphics Processing Units [1]. But even that is now an anachronism, it started dying in 2007 when CUDA was released. With CUDA and OpenCL and compute shaders all being standard, it is now widely understood that today’s GPUs are used for general purpose compute and might not do any graphics. The bulk of chip area is general purpose and has been for some time. From a software development perspective GPU is just a legacy name but is not causing productivity problems or confusion. To be fair, yes most GPUs still do come with things like texture units, video transcode units, ray tracing cores, and a framebuffer and video output. But that’s already changing and you have, for example, some GPUs with ray tracing, and some without that are more designed for data centers. And you don’t have to use the graphics functionality; for GPU supercomputers it’s common for the majority of GPU nodes to be compute-only. In the mean time we now have CPUs with embedded GPUs (aka iGPUs), GPUs with embedded CPUs, GPUs that come paired with CPUs and a wide interconnect (like Nvidia Grace Hopper), CPU-GPU chips (like Apple M1), and yes CPUs in general have more and more SIMD. It’s useful to have a name or a way to distinguish between a processor that mostly uses a single threaded SISD programming model and has a small handful of hardware threads, versus a processor that uses a SIMD/SIMT model and has tens of thousands of threads. That might be mainly a question of workloads and algorithms, but the old line between CPU and GPU is very blurry, headed towards extinction, and the “graphics” part has already lost meaning. [1] https://en.wikipedia.org/wiki/General-purpose_computing_on_graphics_processing_units https://en.wikipedia.org/wiki/General-purpose_computing_on_g...
- adrian_b 2y ago
  The display controller, which handles the frame buffers and the video outputs, and the video decoding/encoding unit are two blocks that are usually well separated from the remainder of the GPU. In many systems-on-a-chip, the 3 blocks, GPU in the strict sense, video decoder/encoder and display controller may even be licensed from different IP vendors and then combined in a single chip. Also in the CPUs with intgrated GPU, like Intel Lunar Lake and AMD Strix Point, these 3 blocks can be found in well separated locations on the silicon die. What belongs into the GPU proper from the graphics-specific functions, because these perform operations that are mixed with the general-purpose computations done by shaders, are the ray-tracing units, the texture units and the rasterization units.

JonChesterfield 2y ago

The x64 cores putting more hardware into the vector units and amdgpu changing from 64 wide to 32 wide simd (at least for some chips) looks like convergent evolution to me. My personal belief is that the speculation and pipelining approach is worse than the many tasks and swapping between them. I think the APU designs from AMD are the transition pointing to the future. The GPU cores will gain increasing access to the raw hardware and the user interface until the CPU cores are optional and ultimately discarded.

adrian_b 2y ago
There is little relationship between the reasons that determine the width of SIMD in CPUs and GPUs, so there is no convergence between them. In the Intel/AMD CPUs, the 512-bit width, i.e. 64 bytes or 16 FP32 numbers, matches the width of the cache line and the width of a DRAM burst transfer, which simplifies the writing of optimized programs. This SIMD width also provides a good ratio between the power consumed in the execution units and the power wasted in the control part of the CPU (around 80% of the total power consumption goes to the execution units, which is much more than when using narrower SIMD instructions). Increasing the SIMD width more than that in CPUs would complicate the interaction with the cache memories and with the main memory, while providing only a negligible improvement in the energy efficiency, so there is no reason to do this. At least in the following decade it is very unlikely that any CPU would increase the SIMD width beyond 16 FP32 numbers per operation. On the other hand, the AMD GPUs before RDNA had a SIMD width of 64 FP32 numbers, but the operations were pipelined and executed in 4 clock cycles, so only 16 FP32 numbers were processed per clock cycle. RDNA has doubled the width of the SIMD execution, processing 32 FP32 numbers per clock cycle. For this, SIMD instructions with a reduced width of 32 FP32 have been introduced, but they are executed in one clock cycle versus the old 64 FP32 instructions that were executed in four clock cycles. For backwards compatibility, RDNA has kept 64 FP32 instructions, which are executed in two clock cycles, but these were not recommended for new programs. RDNA 3 has changed again all this, because now sometimes the 64 FP32 instructions can be executed in a single clock cycle, so they may be again preferable instead of the 32 FP32 instructions. However it is possible to take advantage of the increased width of the RDNA 3 SIMD execution units also when using 32 FP32 instructions, if certain new instructions are used, which encode double operations. So the AMD GPUs have continuously evolved towards wider SIMD execution units, from 16 FP32 before RDNA, to 32 FP32 in RDNA and finally to 64 FP32 in RDNA 3. The distance from CPUs has been steadily increasing, there is no convergence.

Symmetry 2y ago

There are still a lot of differences, even if you put in a lot more SIMD lanes to the CPU. CPUs keep their execution resources fed by by aggressive caching, prefetching, and out of order execution while GPUs rely on having lots of threads around so that if one stalls another is able to execute.

bee_rider 2y ago

Xeon Phi was that, a bunch of little cores with a ton of SIMD lanes each. It didn’t really work out, in part because it was too far from a regular old Xeon to run your normal code well without optimizing. On the other side, Intel couldn’t keep up with NVIDIA on the sort of metrics people care about for these compute accelerators: memory bandwidth mostly. If you are going to have to refactor your whole project anyway to use a compute accelerator, you probably want a pretty big reward. It isn’t obvious (to me at least) if this is the result of the fact that the Phi cores, simple as they were, were still a lot more complex than a GPU “core,” maybe the design just had various hidden bottlenecks that were too hard to work out due to that complexity. Or if it is because Intel just wasn’t executing very well at the time, especially compared to NVIDIA (it is Intel’s dark age vs NVIDIA’s golden age, really). The programmer’s “logical or” joke is a possible answer here. But, you can’t do everything in parallel. It is a shame the Phi didn’t survive into the age where Intel is also doing big/little cores (in a single chip). A big Xeon core (for latency) surrounded by a bunch of little Phi cores (for throughput) could have been a really interesting device.

dahart 2y ago

The special purpose graphics distinction is already mostly irrelevant and has been for 10 or 20 years for anyone doing High Performance Computing (HPC) or AI. It predates LLMs. For a while we had the acronym GPGPU - General Purpose computing on Graphics Processing Units [1]. But even that is now an anachronism, it started dying in 2007 when CUDA was released. With CUDA and OpenCL and compute shaders all being standard, it is now widely understood that today’s GPUs are used for general purpose compute and might not do any graphics. The bulk of chip area is general purpose and has been for some time. From a software development perspective GPU is just a legacy name but is not causing productivity problems or confusion. To be fair, yes most GPUs still do come with things like texture units, video transcode units, ray tracing cores, and a framebuffer and video output. But that’s already changing and you have, for example, some GPUs with ray tracing, and some without that are more designed for data centers. And you don’t have to use the graphics functionality; for GPU supercomputers it’s common for the majority of GPU nodes to be compute-only. In the mean time we now have CPUs with embedded GPUs (aka iGPUs), GPUs with embedded CPUs, GPUs that come paired with CPUs and a wide interconnect (like Nvidia Grace Hopper), CPU-GPU chips (like Apple M1), and yes CPUs in general have more and more SIMD. It’s useful to have a name or a way to distinguish between a processor that mostly uses a single threaded SISD programming model and has a small handful of hardware threads, versus a processor that uses a SIMD/SIMT model and has tens of thousands of threads. That might be mainly a question of workloads and algorithms, but the old line between CPU and GPU is very blurry, headed towards extinction, and the “graphics” part has already lost meaning. [1] https://en.wikipedia.org/wiki/General-purpose_computing_on_graphics_processing_units https://en.wikipedia.org/wiki/General-purpose_computing_on_g...

adrian_b 2y ago
The display controller, which handles the frame buffers and the video outputs, and the video decoding/encoding unit are two blocks that are usually well separated from the remainder of the GPU. In many systems-on-a-chip, the 3 blocks, GPU in the strict sense, video decoder/encoder and display controller may even be licensed from different IP vendors and then combined in a single chip. Also in the CPUs with intgrated GPU, like Intel Lunar Lake and AMD Strix Point, these 3 blocks can be found in well separated locations on the silicon die. What belongs into the GPU proper from the graphics-specific functions, because these perform operations that are mixed with the general-purpose computations done by shaders, are the ray-tracing units, the texture units and the rasterization units.