2 ms·

Do you really need a KV cache if inference is that fast though?

• Tuna-Fish 2mo ago
  ... Yes. Quadratic is really bad for large enough n, and you need that big context for useful work.