3 ms·
Sure, but any classical attention mechanism is quadratic in context length.
by saagarjha 4mo ago
Sure, but any classical attention mechanism is quadratic in context length.
- zozbot234 4mo agoBut text generation is quadratic after the KV cache optimization. If every decode step now has to recompute KV cache including its latest and most expensive tokens (even with a quick, "draft" model) that's even worse.