4 ms·
Compaction has been a pretty painful part of local llm usage. Scrapping the current context and parsing almosy 128k of context then generating something like 5-
by kennywinker 2mo ago
Compaction has been a pretty painful part of local llm usage. Scrapping the current context and parsing almosy 128k of context then generating something like 5-10k tokens - that can take quite a while when you’re working with 10t/s-45t/s (depending on the model).
I pretty much just start a new session whenever i fill the context.
- xyzzy123 2mo agoIn my opinion this is one of the areas where GPUs provide a qualitatively different experience than unified memory boxes. For an EPYC with a 5090 (no layers on CPU) vs an M3 max 128GB, qwen 3.6 27B at 128k context / 7k generation: Cold: prefill + decode Hot (KV cached) 5090 40s + 2-3m = 3-4 min 2-3 min M3 Max 128GB 14m + 8-10m = 22-25 min 8-10 min This is for dense qwen (which I wouldn't run day to day on the mac) - in reality the mac is quite usable with MoEs but you definitely notice a difference.