2 ms·

How does this work with anything but trivially small context sizes!?

• jychang 1y ago
  Tensor parallelism, so you only need to store a fraction of kv cache per gpu.