8 ms·
For local runs, I made some GGUFs! You need around RAM + VRAM >= 250GB for good perf for dynamic 2bit (2bit MoE, 6-8bit rest) - can also do SSD offloading but i
by danielhanchen 1y ago
For local runs, I made some GGUFs! You need around RAM + VRAM >= 250GB for good perf for dynamic 2bit (2bit MoE, 6-8bit rest) - can also do SSD offloading but it'll be slow.
./llama.cpp/llama-cli -hf unsloth/DeepSeek-V3.1-GGUF:UD-Q2_K_XL -ngl 99 --jinja -ot ".ffn_.*_exps.=CPU"
More details on running + optimal params here: https://docs.unsloth.ai/basics/deepseek-v3.1 https://docs.unsloth.ai/basics/deepseek-v3.1
- tw1984 1y agofor such dynamic 2bit, is there any benchmark results showing how many performance I would give up compared to the original model? thanks.
- danielhanchen 1y agoCurrently no, but I'm running them! Some people on the aider discord are running some benchmarks!
- cowpig 1y ago@danielhanchen do you publish the benchmarks you run anywhere?
- danielhanchen 1y agoWe had benchmarks for Llama 4 and Gemma 3 at https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs - for others I normally refer to https://discord.com/channels/1131200896827654144/1282240423661666337 https://discord.com/channels/1131200896827654144/12822404236... which is the Aider Polygot Discord - they always benchmark our quants :)
- segmondy 1y agoif you are running a 2bit quant, you are not giving up performance but gaining 100% performance since the alternative is usually 0%. Smaller quants are for folks who won't be able to run anything at all, so you run the largest you can run relative to your hardware. I for instance often ran Q3_K_L, I don't think of how much performance I'm giving up, but rather how without Q3, I won't be able to run it at all. With that said, for R1, I did some tests against 2 public interfaces and my local Q3 crushed them. The problem with a lot of model providers is we can never be sure what they are serving up and could take shortcuts to maximize profit.
- linuxftw 1y agoThat's true only in a vacuum. For example, should I run gpt-oss-20b unquantized or gpt-oss-120b quantaized? Some models have a 70b/30b spread, and that's only across a single base model, where many different models exist at different quants could be compared for different tasks.
- jkingsman 1y agoDefinitely. As a hobbyist, I have yet to put together a good heuristic for better-quant-lower-params vs. smaller-quant-high-params. I've mentally been drawing the line at around q4, but now with IQ quants and improvements in the space I'm not so sure anymore.
- linuxftw 1y agoYeah, I've kinda quickly thrown in the towel trying to figure out what's 'best' for smaller memory systems. As things are just moving so quickly, whatever time I invest into that is likely to be for nil.
- danielhanchen 1y agoFor GPT OSS in particular, OpenAI only released the MoEs in MXFP4 (4bit), so the "unquantized" version is 4bit MoE + 16bit attention - I uploaded "16bit" versions to https://huggingface.co/unsloth/gpt-oss-120b-GGUF https://huggingface.co/unsloth/gpt-oss-120b-GGUF, and they use 65.6GB whilst MXFP4 uses 63GB, so it's not that much difference - same with GPT OSS 20B llama.cpp also unfortunately cannot quantize matrices that are not a multiple of 256 (2880)
- danielhanchen 1y agoOh Q3_K_L as in upcasted embed_tokens + lm_head to Q8_0? I normally do Q4 embed Q6 lm_head - would a Q8_0 be interesting?
- pshirshov 1y agoBy the way, I'm wondering why unsloth (a goddamn python library) tries to run apt-get with sudo (and fails on my nixos). Like how tf we are supposed to use that?
- danielhanchen 1y agoOh hey I'm assuming this is for conversion to GGUF after a finetune? If you need to quantize to GGUF Q4_K_M, we have to compile llama.cpp, hence apt-get and compiling llama.cpp within a Python shell. There is a way to convert to Q8_0, BF16, F16 without compiling llama.cpp, and it's enabled if you use `FastModel` and not on `FastLanguageModel` Essentially I try to do `sudo apt-get` if it fails then `apt-get` and if all fails, it just fails. We need `build-essential cmake curl libcurl4-openssl-dev` See https://github.com/unslothai/unsloth-zoo/blob/main/unsloth_zoo/llama_cpp.py#L298 https://github.com/unslothai/unsloth-zoo/blob/main/unsloth_z...
- elteto 1y agoDude, this is NEVER ok. What in the world??? A third party LIBRARY running sudo commands? That’s just insane. You just fail and print a nice error message telling the user exactly what they need to do, including the exact apt command or whatever that they need to run.
- pxc 1y agoHow unusual is this for the ecosystem?
- pshirshov 1y agoUnfortunately, Python ecosystem is the Wild West.
- danielhanchen 1y agoYes I had that at the start, but people kept complaining they don't know how to actually run terminal commands, hence the shortcut :( I was thinking if I can do it during the pip install or via setup.py which will do the apt-get instead. As a fallback, I'll probably for now remove shell executions and just warn the user
- zargon 1y agoThanks for your great work with quants. I would really appreciate UD GGUFs for V3.1-Base (and even more so, GLM-4.5-Base + Air-Base).
- danielhanchen 1y agoThanks! Oh base models? Interesting since I normally do only Instruct models - I can take a look though!
- efilife 1y ago>250GB, how do you guys run this stuff?
- danielhanchen 1y agoI'm working on sub 165GB ones! 165GB will need a 24GB GPU + 141GB of RAM for reasonably fast inference or a Mac
- diggan 1y ago> More details on running + optimal params here: https://docs.unsloth.ai/basics/deepseek-v3.1 https://docs.unsloth.ai/basics/deepseek-v3.1 Was that document almost exclusively written with LLMs? I looked at it last night (~8 hours ago) and it was riddled with mistakes, most egregious was that the "Run with Ollama" section had instructions for how to install Ollama, but then the shell commands were actually running llama.cpp, a mistake probably no human would make. Do you have any plans on disclosing how much of these docs are written by humans vs not? Regardless, thanks for the continued release of quants and weights :)
- danielhanchen 1y agoOh hey sorry the docs are still in construction! Are you referring to merging GGUFs to Ollama - it should work fine! Ie: ``` ./llama.cpp/llama-gguf-split --merge \ DeepSeek-V3.1-GGUF/DeepSeek-V3.1-UD-Q2_K_XL/DeepSeek-V3.1-UD-Q2_K_XL-00001-of-00006.gguf \ merged_file.gguf ``` Ollama can only allow merged GGUFs (not splitted ones), so hence the command. All docs are made by humans (primarily my brother and me), just sometimes there might be some typos (sorry in advance) I'm also uploading Ollama compatible versions directly so ollama run can work (it'll take a few more hours)
- wfn 1y ago> but then the shell commands were actually running llama.cpp, a mistake probably no human would make. But in the docs I see things like cp llama.cpp/build/bin/llama-* llama.cpp Wouldn't this explain that? (Didn't look too deep)
- danielhanchen 1y agoYes it's probs the ordering od the docs thats the issue :) Ie https://docs.unsloth.ai/basics/deepseek-v3.1#run-in-llama.cpp https://docs.unsloth.ai/basics/deepseek-v3.1#run-in-llama.cp... does: ``` apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split llama-mtmd-cli llama-server cp llama.cpp/build/bin/llama-* llama.cpp ``` but then Ollama is above it: ``` ./llama.cpp/llama-gguf-split --merge \ DeepSeek-V3.1-GGUF/DeepSeek-V3.1-UD-Q2_K_XL/DeepSeek-V3.1-UD-Q2_K_XL-00001-of-00006.gguf \ merged_file.gguf ``` I'll edit the area to say you first have to install llama.cpp
- azinman2 1y agoIt’d also be great if you guys could do a fine tune to run on an 8x80G A/H100. These H200/B200 configs are harder to come by (and much more expensive).
- danielhanchen 1y agoUnsloth should work on any GPU setup all the way until the old Tesla T4s and the newer B200s :) We're working on a faster and better multi GPU version, but using accelerate / torchrun manually + Unsloth should work out of the box!
- azinman2 1y agoI guess I was hoping for you guys to put up these weights. I think they’d be popular for these very large models. You guys already do a lot for the local LLM community and I appreciate it.
- danielhanchen 1y agoI'll see what I can do :)