6 ms·
I made some GGUFs for those interested in running them at https://huggingface.co/unsloth/Magistral-Small-2506-GGUF https://huggingface.co/unsloth/Magistral-Smal
by danielhanchen 1y ago
I made some GGUFs for those interested in running them at https://huggingface.co/unsloth/Magistral-Small-2506-GGUF https://huggingface.co/unsloth/Magistral-Small-2506-GGUF
ollama run hf.co/unsloth/Magistral-Small-2506-GGUF:UD-Q4_K_XL
or
./llama.cpp/llama-cli -hf unsloth/Magistral-Small-2506-GGUF:UD-Q4_K_XL --jinja --temp 0.7 --top-k -1 --top-p 0.95 -ngl 99
Please use --jinja for llama.cpp and use temperature = 0.7, top-p 0.95!
Also best to increase Ollama's context length to say 8K at least: OLLAMA_CONTEXT_LENGTH=8192 ollama serve &. Some other details in https://docs.unsloth.ai/basics/magistral https://docs.unsloth.ai/basics/magistral
- danielhanchen 1y agoTheir paper https://mistral.ai/static/research/magistral.pdf https://mistral.ai/static/research/magistral.pdf is also cool! They edited GRPO via: 1. Removed KL Divergence 2. Normalize by total length (Dr. GRPO style) 3. Minibatch normalization for advantages 4. Relaxing trust region
- Onavo 1y ago> Removed KL Divergence Wait, how are they computing the loss?
- danielhanchen 1y agoOh it's the KL term sorry - beta * KL ie they set beta to 0. The goal of it was to "force" the model not to stray to far away from the original checkpoint, but it can hinder the model from learning new things
- mjburgess 1y agoIt's just a penalty term that they delete
- trc001 1y agoIt's become trendy to delete it. I say trendy because many papers delete it without offering any proof that it is meaningless
- gyrovagueGeist 1y agoDoes anyone know why they added minibatch advantage normalization (or when it can be useful)? The paper they cite "What matters in on-policy RL" claims it does not lead to much difference on their suite of test problems, and (mean-of-minibatch)-normalization doesn't seem theoretically motivated for convergence to the optimal policy?
- danielhanchen 1y agoTbh I'm unsure as well I took a skim of the paper so if I find anything I'll post it here!
- cpldcpu 1y agoBut this is just the SFT - "distilled" model, not the one optimized with RL, right?
- danielhanchen 1y agoOh I think it's SFT + RL as mentioned in the paper - they said combining both is actually more performant than just RL
- lxe 1y agoThanks for all you do!
- danielhanchen 1y agoThanks!
- monkmartinez 1y agoAt the risk of dating myself; Unsloth is the Bomb-dot-com!!! I use your models all the time and they just work. Thank you!!! What does llama.cpp normally use if not "jinja" for their templates?
- danielhanchen 1y agoOh thanks! Yes I was gonna bring it up to them! Imo if there is a chat template, by default it should be --jinja
- ozgune 1y agoTheir benchmarks are interesting. They are comparing to DeepSeek-V3's (non-reasoning) December and DeepSeek-R1's January releases. I feel that comparing to DeepSeek-R1-0528 would be more fair. For example, R1 scores 79.8 on AIME 2024, R1-0528 performs 91.4. R1 scores 70 on AIME 2025, R1-0528 scores 87.5. R1-0528 does similarly better for GPQA Diamond, LiveCodeBench, and Aider (about 10-15 points higher). https://huggingface.co/deepseek-ai/DeepSeek-R1-0528 https://huggingface.co/deepseek-ai/DeepSeek-R1-0528
- semi-extrinsic 1y agoWould also be interesting to compare with R1-0528-Qwen3-8B (chain-of-thought distilled from Deepseek-R1-0528 and post-trained into Qwen3-8B). It scores 86 and 76 on AIME 2024 and 2025 respectively. Currently running the 6-bit XL quant on a single old RTX 2080 Ti and I'm quite impressed TBH. Simply wild for a sub-8GB download.
- danielhanchen 1y agoI'm surprised it does very well as well - that's pretty cool to see!
- saratogacx 1y agoI have the same card on my machine at home, what is your config to run the model?
- semi-extrinsic 1y agoDownloaded the gguf file by unsloth, ran llama-cli from llama.cpp with that file as an argument. IIUC, nowadays there is a jinja templated metadata-struct inside the gguf file itself. This contains the chat template and other config.
- derefr 1y agoI presume that "outdated upon release" benchmarks like these happen because the benchmark and the models in it were chosen first, before the model was created; and the model's development progress was measured using the benchmark. It then doesn't occur to anyone that the benchmark the engineers had been relying upon isn't also a good/useful benchmark for marketing upon release. From the inside view, it's just a benchmark, already there, already achieving impressive results, a whole-company internal target to hit for months — so why not publish it?
- gavi 1y agotoo much thinking https://gist.github.com/gavi/b9985f730f5deefe49b6a28e5569d467 https://gist.github.com/gavi/b9985f730f5deefe49b6a28e5569d46...
- fzzzy 1y agoMy impression from running the first R1 release locally was that it also does too much thinking.
- cluckindan 1y agoIt does not do any thinking. It is a statistical model, just like the rest of them.
- robmccoll 1y agoWhat are we doing when we think?
- otabdeveloper4 1y agoWe don't know yet. But we do know it's certainly not statistical token prediction. (People can do statistical token prediction too, but that's called "bullshitting", not "thinking". Thinking is a much wider class of activity.)
- LordDragonfang 1y agoDo we know that with certainty? Do we actually? Because my understanding is that how "thinking" works is actually still a total mystery. How is it we no for certain that the basis for the analog electric-potential-based computing done by neurons is not based on statistical prediction? Do we have actual evidence of that, or are you just doing "statistical token prediction" yourself?
- cluckindan 1y ago
- trebligdivad 1y agoNice! I'm running on CPU only, so it's interesting to compare - the Magistral-Small-2506_Q8_0.gguf runs at under 2 tokens/s on my 16 core, but your UD-IQ2_XXS gets about 5.5 tokens/s which is fast enough to be useful - but it does hallucinate a bit more and loop a little; but still actually pretty good for something so small.
- danielhanchen 1y agoOh nice! I normally suggest maybe Q4_K_XL to be on the safe side :)