You have to load the model weights into VRAM over PCI-E (from RAM). So the (PCI-E) bandwidth strongly affects time to first token. You have to run inference on

You have to load the model weights into VRAM over PCI-E (from RAM). So the (PCI-E) bandwidth strongly affects time to first token. You have to run inference on the GPU by reading and writing to VRAM. So TFLOPS of the compute matters, and bandwidth to the VRAM (Always integrated with the GPU, rarely a bottleneck), and this strongly affects tokens/s If you're doing training workloads or offloading to system RAM, it gets more complicated. (And mostly bound up trying to feed compute on time) (Edits for clarity.)