3 ms·
Oh we actually fixed bugs! We fixed a few bugs in Gemma - see https://news.ycombinator.com/item?id=39671146 https://news.ycombinator.com/item?id=39671146, a gra
by danielhanchen 1y ago
Oh we actually fixed bugs! We fixed a few bugs in Gemma - see https://news.ycombinator.com/item?id=39671146 https://news.ycombinator.com/item?id=39671146, a gradient accumulation bug see https://news.ycombinator.com/item?id=41859037 https://news.ycombinator.com/item?id=41859037, Phi bugs, Llama bugs and more! See https://unsloth.ai/blog/reintroducing https://unsloth.ai/blog/reintroducing for more details!
- menaerus 1y agoWhat does your approach with dynamics weights has to do with those bugs? All those bugs seem uncorrelated to the technique.
- danielhanchen 1y agoOh apologies I got confused - it's because when we calculate our dynamic quants, we have to do it on the fixed model! For example in Phi 3 for example, the end of sentence token was wrong - if we use this, then our quants would be calibrated incorrectly, since chatting with the model will use the actual correct token. Another is Llama 4 - https://github.com/ggml-org/llama.cpp/pull/12889 https://github.com/ggml-org/llama.cpp/pull/12889 in which I fixed a RoPE issue - if we didn't fix it first, then again the calibration process would be incorrect.
- menaerus 1y agoOk, this then goes to say that your approach doesn't work without applying whatever fixes to the vanilla models. What I'm trying to understand is the approach itself. Why does it and how does it work?
- danielhanchen 1y agoOh I wrote a bit about it in https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs and https://unsloth.ai/blog/deepseekr1-dynamic https://unsloth.ai/blog/deepseekr1-dynamic if that helps!