3 ms·
I recently wrote a guide on getting: - llama.cpp - OpenCode - Qwen3-Coder-30B-A3B-Instruct in GGUF format (Q4_K_M quantization) working on a M1 MacBook Pro
by alexpotato 8mo ago
I recently wrote a guide on getting:
- llama.cpp
- OpenCode
- Qwen3-Coder-30B-A3B-Instruct in GGUF format (Q4_K_M quantization)
working on a M1 MacBook Pro (e.g. using brew).
It was bit finicky to get all of the pieces together so hopefully this can be used with these newer models.
https://gist.github.com/alexpotato/5b76989c24593962898294038b5b835b https://gist.github.com/alexpotato/5b76989c24593962898294038...
- copperx 8mo agoHow fast does it run on your M1?
- robby_w_g 8mo agoDoes your MBP have 32 GB of ram? I’m waiting on a local model that can run decently on 16 GB
- freeone3000 8mo agoWe can also run LM Studio and get it installed with one search and one click, exposed through an OpenAI-compatible API.
- kpw94 8mo agoOn my 32GB Ryzen desktop (recently upgraded from 16GB before the RAM prices went up another +40%), did the same setup of llama.cpp (with Vulkan extra steps) and also converged on Qwen3-Coder-30B-A3B-Instruct (also Q4_K_M quantization) On the model choice: I've tried latest gemma, ministral, and a bunch of others. But qwen was definitely the most impressive (and much faster inference thanks to MoE architecture), so can't wait to try Qwen3.5-35B-A3B if it fits. I've no clue about which quantization to pick though ... I picked Q4_K_M at random, was your choice of quantization more educated?
- NamlchakKhandro 8mo ago[flagged]
- zargon 8mo agoQuant choice depends on your vram, use case, need for speed, etc. For coding I would not go below Q4_K_M (though for Q4, unsloth XL or ik_llama IQ quants are usually better at the same size). Preferably Q5 or even Q6.