CPUs and GPUs both interface with their own memory, and those memories have a certain bandwidth. Generally, CPU memory has relatively little bandwidth, but rela

CPUs and GPUs both interface with their own memory, and those memories have a certain bandwidth. Generally, CPU memory has relatively little bandwidth, but relatively good latency. (For example, an i9-13900K supports memory up to around 100 GB/s, while even my previous GPU, a midrange Radeon HD 7850 from 2012, has over 150 GB/s of bandwidth). An integrated GPU shares memory with the CPU, so at best it gets the same amount of bandwidth assuming the CPU is not using any (which is rather unlikely). A dedicated GPU has its own private high-bandwidth memory (an RTX 4090 has a memory bandwidth of over 1000 GB/s), but to get anything in there it needs to be loaded over the PCIe bus (which has a measly 32 GB/s bandwidth for PCIe 4.0 x16). That said, CPU memory does have one big advantage: it tends to be much larger. You can pair up to 192 GB with a regular Ryzen 7000 CPU, while consumer GPUs don't go above 24 GB of memory (RTX 4090, RX 7900 XTX). (There are bigger GPUs out there, but those are generally intended for datacenters, and if you go that route, an Epyc or Xeon CPU can also support much more memory than a plain desktop Ryzen, although you can also slot multiple GPUs into a single server.) I believe that for LLM performance, memory bandwidth is key, because all the neural network layers need to be streamed from memory, and very little work is done with it each time, although I guess batching operations could help if you're working at scale, since each weight would be applied multiple times then.