3 ms·
I had been unaware of the others. Anyway, you need writes to the KV cache for every token generated. You are going to hit that fast.
by ryao 1y ago
I had been unaware of the others. Anyway, you need writes to the KV cache for every token generated. You are going to hit that fast.