3 ms·

This is not a valid argument. TPS is essentially QoS and can be adjusted; more GPUs allocated will result in higher speed.

• yorwba 7mo ago
  There are sequential dependencies, so you can't just arbitrarily increase speed by parallelizing over more GPUs. Every token depends on all previous tokens, every layer depends on all previous layers. You can arbitrarily slow a model down by using fewer, slower GPUs (or none at all), though.
  • erichocean 7mo ago
    Partially true, you can predict multiple tokens and confirm, which typically gives a 2-3x speedup in practice. (Confirmation is faster than prediction.) Many models architectures are specifically designed to make this efficient. --- Separately, your statement is only true for the same gen hardware, interconnects, and quantization.
  • grumpoholic 7mo ago
    With speculative decoding you can use more models to speed up the generation however.
    • salawat 7mo ago
      Yes, because speculation has NEVER bitten us in the ass before, right? Coughs in Spectre Speculative decoding is just running more hardware to get a faster prediction. Essentially, setting more money on fire if you're being billed per token.