4 ms·
Nvidia releases 8B model with learned 8x KV cache compression
- alecco 9mo agoThe paper behind it was presented in December 2025 Neurips Release thread: https://xcancel.com/p_nawrot/status/2014770473289019709 https://xcancel.com/p_nawrot/status/2014770473289019709 Slides and audio presentation: https://neurips.cc/virtual/2025/loc/san-diego/poster/119605 https://neurips.cc/virtual/2025/loc/san-diego/poster/119605
- vercaemert 9mo agoI'd be interested to hear some use cases people have for large contexts on an 8B model. Other than sentiment analysis or summarization (this release implies agentic use). My experience with the general intelligence of agentic interactions is that everything is unusable before 32B for any context greater than 4k tokens.
- deleted 9mo ago[deleted]
- WaalkTheEaarth 9mo agoI personally use a 8B model for general use on my laptop lol, it works like a charm and makes sense (most of the time atleast)
- g023 9mo agoI made a smaller sized version https://huggingface.co/g023/Qwen3-8B-DMS-8x-4bit-NF4 https://huggingface.co/g023/Qwen3-8B-DMS-8x-4bit-NF4