3 ms·
That's not true, the GGUF quants aren't great but there are exl2 4bit quantizations floating around that are pretty sweet.
by CuriouslyC 3y ago
That's not true, the GGUF quants aren't great but there are exl2 4bit quantizations floating around that are pretty sweet.
- d-z-m 3y agofair enough, yeah I'm talking about GGUF quants only.
- brucethemoose2 3y agoexl2 is Nvidia/AMD only. But GGUF Mixtral should fit in 32GB... just not with the full 32K context. Long context is very memory intense in llama.cpp, at least until they fully implement flash attention and a quantized cache.