7 ms·
Looks impressive, and this size fits achievable home hardware. That said, if someone would kindly quantise this down for the 64GB paupers, that would be apprec
by mft_ 3mo ago
Looks impressive, and this size fits achievable home hardware.
That said, if someone would kindly quantise this down for the 64GB paupers, that would be appreciated. (I know there’s likely degradation, but some people reported good results with a 2 bit version of Qwen 3.5 122B, and this is starting from a higher point. Would be interesting to try, at least.)
Edit: someone in the process of doing so: https://huggingface.co/vcruz305/Laguna-S-2.1-GGUF https://huggingface.co/vcruz305/Laguna-S-2.1-GGUF
- verdverm 3mo agoThe tool I've been using, llm-compressor, can quant models that do not fit in memory (use the sequential pipeline) https://github.com/vllm-project/llm-compressor https://github.com/vllm-project/llm-compressor my setup to help you on your way: https://github.com/verdverm/quantr https://github.com/verdverm/quantr Though it seems these will not be needed as Poolside has published quants & dflash with their models.
- alfiedotwtf 3mo agoNice! Do you know of any tools that do this for tensorrt models?
- verdverm 3mo agoSorry, I don't, still newer to the quantizing side of things since many are already produced by others.
- yogeshp 3mo agoThey have also published smaller 33B model called Laguna XS 2.1, its Q4 gguf is 20GB. https://huggingface.co/poolside/Laguna-XS-2.1-GGUF/tree/main https://huggingface.co/poolside/Laguna-XS-2.1-GGUF/tree/main
- mft_ 3mo agoThanks for flagging. From the few benchmarks I can find, it looks there or thereabouts with Qwen 3.6-35B-A3B, or maybe a touch below. I'm interested to compare a model that is a big jump larger with pretty impressive benchmarks, but more heavily quantized to fit.
- dofm 3mo agoThe XS model is genuinely confounding. So far I do not know what to make of it. Maybe the chat template is broken, maybe not. But I can't get it to think/reason, or at least sometimes it seems to decide not to even when asked. And it is wildly inconsistent when it does it, sometimes thinking out loud. However, without thinking enabled it is shockingly good at researching using web tool calls. It's also very fast. I am assuming something is not yet right with the chat template, even using a recent build of llama.cpp, and I guess I will revisit this model again at some point in the future.
- Archit3ch 3mo ago> if someone would kindly quantise this down for the 64GB paupers The Q4_K_M is 75GB. I'm exactly at 64GB, and I wouldn't quantize it further. Instead, do partial weight residency and stream the rest from SSD.
- bitexploder 2mo agoI am going to see what I can get out of this streaming from SSD vs. Qwen 27B. Laguna is a seriously impressive model. I already have a lot of success with 27B A3B.
- nunodonato 3mo agounsloth is always here to save the day https://huggingface.co/unsloth/Laguna-S-2.1-GGUF https://huggingface.co/unsloth/Laguna-S-2.1-GGUF
- andy99 3mo agoSeems it’s not fully supported in mainline llama.cpp yet https://github.com/ggml-org/llama.cpp/pull/25165 https://github.com/ggml-org/llama.cpp/pull/25165 In the huggingface link they mention building for CPU and for CUDA, does anyone know if that means it wouldn’t be possible to build targeting Vulkan?
- andy99 3mo agoReplying to myself, seems this PR was merged into main and it the model does work with a Vulkan backend on my Framework desktop, I’m getting about 220 tok/s prompt processing and 21 tok/s output on the 4-bit quant. This is really a sweet spot imo on this machine between maximizing ram use and still having decent speed due to the expert size. This looks really promising.
- regularfry 3mo agoWhat's your hardware? I've got 64GB RAM and a 4090 here, wondering if it's worth a play.
- andy99 3mo agoIt’s the Strix halo (AMD) with 128 GB shared memory. The 4bit quant is ~75GB. Unfortunately I don’t know about the best way of running on an Nvidia gpu, you could try llama.cpp and offloading as many layers as possible into the gpu and using RAM for the rest, not sure if that would slow it down too much though.
- regularfry 3mo agoI got usable token rates (10-20tps from memory, so marginal) with the Qwen A10B a while back, well before all the new speculative speedups landed in llama.cpp. There's an unmerged branch which allegedly supports this, but I don't know how well yet. Looks worth investigating but I might give it a few days to see what bugs get shaken loose.
- mft_ 3mo agoThis is interesting: https://github.com/tanishq-dubey/macos-laguna-s2.1 https://github.com/tanishq-dubey/macos-laguna-s2.1 Someone has benchmarked a wide range of different quants.