You have to get those weights from the RAM to the floating point unit. The bandwidth here is the rate at which you can do that. The weights are not really reus

You have to get those weights from the RAM to the floating point unit. The bandwidth here is the rate at which you can do that. The weights are not really reused. Which means they are never in registers, or in L1/L2/L3 caches. They are always in VRAM and always need to be loaded back in again. However, if you are batching multiple separate inputs you can reuse each weight on ech input, in which case you may not be entirely bandwidth bound and this analysis breaks down a bit. Basically you can't produce a single stream of tokens any faster than this rate, but you can produce more than one stream of tokens at this rate.