3 ms·
GPUs might not be bandwidth starved most of the time, but they absolutely are when generating text from an llm. It’s the whole reason why low precision floating
by djsjajah 10mo ago
GPUs might not be bandwidth starved most of the time, but they absolutely are when generating text from an llm.
It’s the whole reason why low precision floating point numbers are being pushed by nvidia.
- ACCount37 10mo agoThat's memory bandwidth, not I/O. Unless your LLM doesn't fit into VRAM.