3 ms·

So am I correct in understanding what they really mean is 37 full forward passes per second? In which case, if the model weights are fitting in the VRAM and ar

So am I correct in understanding what they really mean is 37 full forward passes per second? In which case, if the model weights are fitting in the VRAM and are already loaded, why does the bandwidth impact the rate of tok/s?

fancyfredbot 2y ago
You have to get those weights from the RAM to the floating point unit. The bandwidth here is the rate at which you can do that. The weights are not really reused. Which means they are never in registers, or in L1/L2/L3 caches. They are always in VRAM and always need to be loaded back in again. However, if you are batching multiple separate inputs you can reuse each weight on ech input, in which case you may not be entirely bandwidth bound and this analysis breaks down a bit. Basically you can't produce a single stream of tokens any faster than this rate, but you can produce more than one stream of tokens at this rate.