4 ms·
I'm making dynamic GGUFs for local inference at https://huggingface.co/unsloth/Qwen3-235B-A22B-Thinking-2507-GGUF https://huggingface.co/unsloth/Qwen3-235B-A22B
by danielhanchen 1y ago
I'm making dynamic GGUFs for local inference at https://huggingface.co/unsloth/Qwen3-235B-A22B-Thinking-2507-GGUF https://huggingface.co/unsloth/Qwen3-235B-A22B-Thinking-2507... Also guide to run them https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507 https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tun...
- arcanemachiner 1y agoWould I notice a difference between the Q2_K and Q2_K_XL variants?
- danielhanchen 1y agoOh I would always use Q2_K_XL :) It uses our dynamic methodology to quantize certain layers in different bits ie 2, 3, 4, 5, 6, 8 bits - the more important the layer is, the higher the bitrate
- Squeeze2664 1y agoHow do you determine the importance of a layer in this case?
- kkzz99 1y agoAfaik they have a test bench that they use and take the activation data from that.
- danielhanchen 1y agoYes we have around 1 to 3 million tokens of high quality self verified data that we use to calibrate models!
- smallerize 1y agohttps://unsloth.ai/blog/dynamic-v2 https://unsloth.ai/blog/dynamic-v2
- danielhanchen 1y agoYes also https://unsloth.ai/blog/deepseekr1-dynamic https://unsloth.ai/blog/deepseekr1-dynamic, https://unsloth.ai/blog/dynamic-4bit https://unsloth.ai/blog/dynamic-4bit, https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs
- DrPhish 1y agoI generally download the safetensors and make my own GGUFs, usually at Q8_0. Is there any measurable benefit to your dynamic quants at that quant level? I looked at your dynamic quant 2.0 page, but all the charts and graphs appear to cut off at Q4.
- danielhanchen 1y agoOh I also upload Q8_K_XL for eg, which will upcast important layers to BF16 / F16 as well! Oh the blog at https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs does talk about 1, 2, 3, 4, 5, 6 and 8bit dynamic GGUFs as well! There definitely is a benefit for dynamically selecting layers to be at diff bit rates - I wrote about the difference between naively quantizing and selectively quantizing: https://unsloth.ai/blog/deepseekr1-dynamic https://unsloth.ai/blog/deepseekr1-dynamic
- DrPhish 1y agoThanks Daniel. I know you upload them, but I was hoping for some solid numbers on your dynamic q8 vs a naive quant. There doesn't seem to be anything on either of those links to show improvement at those quant levels. My gut feeling is that there's not enough benefit to outweigh the risk of putting a middleman in the chain of custody from the original model to my nvme. However, I can't know for sure without more testing than I have the time or inclination for, which is why I was hoping there had been some analysis you could point me to.
- mycpuorg 1y ago@danielhanchen, can we use these steps to fine tune other qwen3 models too? like 480B coder or embeddings model?
- danielhanchen 1y agoOh for finetuning - we do have some code for MoE finetuning for Qwen at https://github.com/unslothai/unsloth https://github.com/unslothai/unsloth, but we haven't yet announced it yet!
- aliljet 1y agoI see the term 'local inference' everywhere. It's an absurd misnomer without hardware and cost defined. I can also run a coal fired power plant in my backyard, but in practice, there's no reasonable way to make that economical beyond being a toy. (And I should add, you are a hero for doing this work, only love in my comment, but still a demand for detail$!)
- regularfry 1y agoHardware and cost is assumed to be approximately desktop-class. If you've got a gaming rig with an RTX 4090 and 128MB RAM, you can run this if you pick the right quant.
- cmpxchg8b 1y ago128MB? Quantization has come a long way!
- danielhanchen 1y agoI think they mis-spoke 128GB* :)
- regularfry 1y agoWishful thinking there on my part.
- danielhanchen 1y agoThough technically < 1GB is enough - one had offload it to the SSD, albeit with very slow speeds!
- danielhanchen 1y agoThe trick of llama.cpp and our dynamic quants is you can actually offload the model to RAM / even an SSD! If you have GPU VRAM + RAM + SSD > the model size (say 90GB for dynamic 2bit quant), then it'll run well! Ie you can actually run it on a local desktop or even your laptop now! You don't need a 90GB GPU for example, but say a 24GB GPU + 64GB to 128GB RAM. The speeds are around 3 to 5 tokens / second, so still ok! I write more about improving speed for local devices here: https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune/qwen3-2507#improving-generation-speed https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tun...
- lostmsu 1y agoHow usable are 1 and less than 1 bit quants?
- danielhanchen 1y agoOh 1bit reminder isn't 1bit, but a mixture of 1 to 8bit! I would still use 2bit dynamic - 1bit dynamic sometimes can go into weird repetitive loops, but it still does produce reasonable output. Larger models with 1bit do better - for eg 480B Coder 1bit actually does very well!