3 ms·Tensor parallelism, so you only need to store a fraction of kv cache per gpu.by jychang 1y agoTensor parallelism, so you only need to store a fraction of kv cache per gpu.