2 ms·

Is token rate a function of parameter size?

• rbanffy 1mo ago
  Not quite linear, but yes.
• unrahul 1mo ago
  Yes, a quick back of the envelope math is 0.65 * (memory bandwidth of the card / (model weights in bytes + kv cache in bytes) ~ practical decode tps. Below context around 32k (depends upon the model but again can be used as a placeholder number) you can ignore the kv cache in bytes and the math becomes just about memory bandwidth and model weights in bytes.
• giantrobot 1mo ago
  Parameter size and total number of parameters so ultimately the total size of the model in memory. This leads to some interesting optimizations. You can quantize all the parameters (or certain layers) of a model and halve or quarter the memory requirement but maintain most of the model's intelligence. This increases the token rate inversely with the size reduction. Popular quantizations for local models are 8-bit and 4-bit parameter sizes. The Blackwell series of nVidia chips now even support native FP4 math making 4-bit quantizations even faster.
• jjtheblunt 1mo ago
  if you think of the depth of a model increasing with parameter count, then the number of steps of parallel operations to percolate data through a model would increase with parameter count, which would be inversely related to rate of computing tokens as a result of flowing data through the model.