5 ms·

I'm surprised at the simplicity of the formula in the paragraph below. Could someone explain the relationship between model size, memory bandwidth and token/s a

I'm surprised at the simplicity of the formula in the paragraph below. Could someone explain the relationship between model size, memory bandwidth and token/s as they calculated here? > Taking LLaMA 3 70B as an example, in float16 the weights are approximately 140GB, and the generation context adds another ~2GB. MI300X’s theoretical maximum is 5.3TB/second, which gives us a hard upper limit of (5300 / 142) = ~37.2 tokens per second.

wmf 2y ago
AFAIK generating a single token requires reading all the weights from RAM. So 5300 GB/s total memory bandwidth / 142 GB weights = ~37.2 tokens per second.
latchkey 2y ago
From Cheese (they don't have a HN account, so I'm posting for them): Each weight is a FP16 float which is 2 Bytes worth of data, you have 70B tokens, so the total amount of data the weights take up is 140GB then you have a couple extra GBs for the context. Then to figure out the theoretical tokens per second you just divide the amount of memory bandwidth, 5300GB/s in MI300X's case, by the amount of data that the tokens and context take up so 5300/142 which is about 37 tokens per second.
- rbanffy 2y ago
  37 somethings per second doesn’t sound fast at all. You need to remember it’s 37 ridiculously difficult things per second.
- omneity 2y ago
  So am I correct in understanding what they really mean is 37 full forward passes per second? In which case, if the model weights are fitting in the VRAM and are already loaded, why does the bandwidth impact the rate of tok/s?
  - fancyfredbot 2y ago
    You have to get those weights from the RAM to the floating point unit. The bandwidth here is the rate at which you can do that. The weights are not really reused. Which means they are never in registers, or in L1/L2/L3 caches. They are always in VRAM and always need to be loaded back in again. However, if you are batching multiple separate inputs you can reuse each weight on ech input, in which case you may not be entirely bandwidth bound and this analysis breaks down a bit. Basically you can't produce a single stream of tokens any faster than this rate, but you can produce more than one stream of tokens at this rate.
immibis 2y ago
That would be higher with batching, right? (5300 / 144) * 2 = ~73.6 and so on.

wmf 2y ago

AFAIK generating a single token requires reading all the weights from RAM. So 5300 GB/s total memory bandwidth / 142 GB weights = ~37.2 tokens per second.

latchkey 2y ago

From Cheese (they don't have a HN account, so I'm posting for them): Each weight is a FP16 float which is 2 Bytes worth of data, you have 70B tokens, so the total amount of data the weights take up is 140GB then you have a couple extra GBs for the context. Then to figure out the theoretical tokens per second you just divide the amount of memory bandwidth, 5300GB/s in MI300X's case, by the amount of data that the tokens and context take up so 5300/142 which is about 37 tokens per second.

rbanffy 2y ago
37 somethings per second doesn’t sound fast at all. You need to remember it’s 37 ridiculously difficult things per second.
omneity 2y ago
So am I correct in understanding what they really mean is 37 full forward passes per second? In which case, if the model weights are fitting in the VRAM and are already loaded, why does the bandwidth impact the rate of tok/s?
- fancyfredbot 2y ago
  You have to get those weights from the RAM to the floating point unit. The bandwidth here is the rate at which you can do that. The weights are not really reused. Which means they are never in registers, or in L1/L2/L3 caches. They are always in VRAM and always need to be loaded back in again. However, if you are batching multiple separate inputs you can reuse each weight on ech input, in which case you may not be entirely bandwidth bound and this analysis breaks down a bit. Basically you can't produce a single stream of tokens any faster than this rate, but you can produce more than one stream of tokens at this rate.

immibis 2y ago

That would be higher with batching, right? (5300 / 144) * 2 = ~73.6 and so on.