3 ms·
This is 128B dense though. the K/V cache on long context is going to be massive
by 2001zhaozhao 6mo ago
This is 128B dense though. the K/V cache on long context is going to be massive
- Havoc 6mo agoDon’t think kv size correlates to dense/moe
- zozbot234 6mo agoKV size correlates with attention parameters which are a subset of active parameters. So a typical MoE model will have way lower KV size than a dense model of equal total parameter count.
- syntaxing 6mo agoWith turbo quant, you would reduce it by over 6X.