3 ms·
It's very hard to find right now but I'm sure they said they don't quantize KV cache, but their weights are in fp8.
by boroboro4 1y ago
It's very hard to find right now but I'm sure they said they don't quantize KV cache, but their weights are in fp8.