3 ms·
RAM is (often) the bottleneck for highly parallel GPUs, but not for CPUs. Though the distinction between the two categories is blurring.
by webmaven 2y ago
RAM is (often) the bottleneck for highly parallel GPUs, but not for CPUs.
Though the distinction between the two categories is blurring.
- ryao 2y agoMemory bandwidth is the bottleneck for both when running GEMV, which is the main operation used by token generation in inference. It has always been this way.