5 ms·
Do 2bit quantizations really work? All the ones I've seen/tried were completely broken even when 4bit+ quantizations worked perfectly. Even if it works for thes
by gnulinux 1y ago
Do 2bit quantizations really work? All the ones I've seen/tried were completely broken even when 4bit+ quantizations worked perfectly. Even if it works for these extremely large models, is it really much better than using something slightly smaller on 4 or 5 bit quant?
- danielhanchen 1y agoOh the Unsloth dynamic ones are not 2bit at all - it's a mixture of 2, 3, 4, 5, 6 and sometimes 8bit. Important layers are in 8bit, 6bit. Less important ones are left in 2bit! I talk more about it here: https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs
- CMCDragonkai 1y agoHow do you decide which layers are the important ones?
- danielhanchen 1y agoI wrote approximately in the blog about it and linked some papers! I also wrote about it here - https://unsloth.ai/blog/dynamic-4bit https://unsloth.ai/blog/dynamic-4bit - one has to inspect the activation and weight quantization errors!
- blensor 1y agoSo you are basically looking at "fMRI" of the "brain" while it's doing a wide range of things and cutting out the things that stay dark the most?
- danielhanchen 1y agoOh that's a good analogy! Yes that sounds right!
- menaerus 1y ago> The key reason to use Unsloth quants is because of our deep involvement in fixing critical bugs across major models sounds convincing, eh ... /s On the less cynical note, approach does look interesting but I'd also like to understand how and why does it work, if it works at all.
- danielhanchen 1y agoOh we actually fixed bugs! We fixed a few bugs in Gemma - see https://news.ycombinator.com/item?id=39671146 https://news.ycombinator.com/item?id=39671146, a gradient accumulation bug see https://news.ycombinator.com/item?id=41859037 https://news.ycombinator.com/item?id=41859037, Phi bugs, Llama bugs and more! See https://unsloth.ai/blog/reintroducing https://unsloth.ai/blog/reintroducing for more details!
- menaerus 1y agoWhat does your approach with dynamics weights has to do with those bugs? All those bugs seem uncorrelated to the technique.
- danielhanchen 1y agoOh apologies I got confused - it's because when we calculate our dynamic quants, we have to do it on the fixed model! For example in Phi 3 for example, the end of sentence token was wrong - if we use this, then our quants would be calibrated incorrectly, since chatting with the model will use the actual correct token. Another is Llama 4 - https://github.com/ggml-org/llama.cpp/pull/12889 https://github.com/ggml-org/llama.cpp/pull/12889 in which I fixed a RoPE issue - if we didn't fix it first, then again the calibration process would be incorrect.
- menaerus 1y agoOk, this then goes to say that your approach doesn't work without applying whatever fixes to the vanilla models. What I'm trying to understand is the approach itself. Why does it and how does it work?
- blensor 1y agoNot an AI researcher here so this is probably common knowledge for people in this field, but I saw a video about the quantization recently and wondered exactly about that, if it's possible to compress a net by using more precision where it counts and less precision where it's not important. And also wondered how one would go about deciding which parts count and which don't Great to know that this is already a thing and I assume model "compression" is going to be the next hot topic
- danielhanchen 1y agoYes you're exactly thinking correctly! We shouldn't quantize a model naively to 2bit or 4bit, but we should do it smartly!
- qxfys 1y agoHow do you pick which one should be 2, which one should be 4, etc. Is this secret sauce? or, something open?
- danielhanchen 1y agoOh I wrote about it here: https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs We might provide some scripts for them in the future!
- qxfys 1y agoThanks! But, I can't find any details on how you "intelligently adjust quantization for every possible layer" from that page. I assume this is a secret? I am wondering about the possibility that different use cases might require different "intelligent quantization", i.e., quantization for LLM for financial analysis might be different from LLM for code generation. I am currently doing a postdoc in this. Interested in doing research together?
- danielhanchen 1y ago
- PeterStuer 1y agoIf you don't mind divulging, what resources and time did it take to dynamically quantize Qwen3-Coder?
- danielhanchen 1y agoIt takes a few hours to compute the imatrix on some calibration dataset since we use more than 1-3 million tokens of high quality data. Then we have to decide on which layers to quantize to higher bits or not, which takes more time. And the quantization creation also takes some hours. Uploading also takes some time as well! Overall 8 hours maybe minimum?
- jychang 1y agoWhat cluster do you have to do the quantizing? I'm guessing you're not using a single machine with a 3090 in your garage.
- danielhanchen 1y agoOh definitely not! I use some spot cloud instances!
- sleight42 1y agoBut you can get one of these quantized models to run effectively on a 3090? If so, I'd love detailed instructions. The guide you posted earlier goes over my (and likely many others') head!
- danielhanchen 1y agoOh yes definitely! Oh wait is the guide too long / wordy? This section https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#llama.cpp-run-qwen3-tutorial https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locall... shows how to run it on a 3090
- tomdekan 1y agoThanks Daniel. Do you recommend any resources showing the differences between different quantizations?
- danielhanchen 1y agoOh our blog https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs compares the accuracy differences for each different quantization method for Llama 4 Scout and also Gemma 3 27B - they should apply to other quants (like Qwen 3 Coder)