3 ms·
It’s interesting to me that my CPU can run some of these things in quantized form almost as fast as the GPU. Has the whole thing been all about memory bandwidth
by api 3y ago
It’s interesting to me that my CPU can run some of these things in quantized form almost as fast as the GPU. Has the whole thing been all about memory bandwidth all along?
In addition to compute the GPU architecture is one that somewhat colocates working memory alongside compute. Units have local memories that sync with global memory. Is that a big part of why GPUs are so good for this?
- brucethemoose2 3y ago> Has the whole thing been all about memory bandwidth all along Yeah, sort of. LLMs like llama at a batch size of 1 are hilariously bandwidth bound. Stable Diffusion less so. Its still bandwidth heavy on GPUs, but compute is much more of a bottleneck.