3 ms·
I got this running locally using llama.cpp from Homebrew and the Unsloth quantized model like this: brew upgrade llama.cpp # or brew install if you don't hav
by simonw 9mo ago
I got this running locally using llama.cpp from Homebrew and the Unsloth quantized model like this:
brew upgrade llama.cpp # or brew install if you don't have it yet
Then:
llama-cli \
-hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_XL \
--fit on \
--seed 3407 \
--temp 1.0 \
--top-p 0.95 \
--min-p 0.01 \
--top-k 40 \
--jinja
That opened a CLI interface. For a web UI on port 8080 along with an OpenAI chat completions compatible endpoint do this:
llama-server \
-hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_XL \
--fit on \
--seed 3407 \
--temp 1.0 \
--top-p 0.95 \
--min-p 0.01 \
--top-k 40 \
--jinja
It's using about 28GB of RAM.
- technotony 9mo agowhat are your impressions?
- simonw 9mo agoI got Codex CLI running against it and was sadly very unimpressed - it got stuck in a loop running "ls" for some reason when I asked it to create a new file.
- danielhanchen 9mo agoYes sadly that sometimes happens - the issue is Codex CLI / Claude Code were designed for GPT / Claude models specifically, so it'll be hard for OSS models directly to utilize the full spec / tools etc, and might get loops sometimes - I would maybe try the MXFP4_MOE quant to see if it helps, and maybe try Qwen CLI (was planning to make a guide for it as well) I guess until we see the day OSS models truly utilize Codex / CC very well, then local models will really take off
- mhitza 9mo agoI would recommend you fiddle with the repeat penalty flags. I use local models often, and almost all I've tried needed that to prevent loops. I'd also recommend dropping temperature down to 0. Any high temperature value feels like instructing the model "copy this homework from me but don't make it obvious".
- CamperBob2 9mo agoYou probably have seen it by now, but there was a llama.cpp issue that was fixed earlier today(?) to avoid looping and other sub-par results. Need to update llama-server as well as redownload the GGUFs (for certain quants). https://old.reddit.com/r/unsloth/comments/1qvt6qy/qwen3codernext_ggufs_updated_now_produces_much/ https://old.reddit.com/r/unsloth/comments/1qvt6qy/qwen3coder...
- simonw 9mo agoI hadn't seen that, thanks very much!
- nubg 9mo agowhat's the token per seconds speed?