4 ms·
For each token generated, you only send one token’s worth between layers; the previous tokens are in the KV cache.
by remexre 10mo ago
For each token generated, you only send one token’s worth between layers; the previous tokens are in the KV cache.