3 ms·
Memory bandwidth is the bottleneck for LLM inference. That's my understanding at least.
by hmottestad 2y ago
Memory bandwidth is the bottleneck for LLM inference. That's my understanding at least.
- moffkalast 2y agoThat's correct, but also compute to some degree. The larger the model the more of a bottleneck memory becomes. There are some older HBM cards with very high bandwidth like the Radeon Pro VII which has 1TB/s of bandwidth like the RTX 3090 and 4090, but is notably slower at inference for smaller models since has less compute in comparison. At least I think that was the consensus of some benchmarks people ran.
- littlestymaar 2y agoIsn't it only the case when inference isn't batched?
- Tostino 2y agoEven in a local setting, batched inference is useful to be able to run more "complex" workflows (with multiple, parallel LLM calls for a single interaction). There is very little reason to optimize for just single stream inference at the expense of your batch inference performance.