3 ms·
Llama.cpp with automatic offload to main memory. You can also use Ollama, it is easier, but slower.
by canpan 5mo ago
Llama.cpp with automatic offload to main memory. You can also use Ollama, it is easier, but slower.
- reverius42 5mo agoFor those who want a GUI, LM Studio does this too (with llama.cpp as the backend I think). I'm getting great (albeit slow) results with Qwen3.6-35B MoE on 8GB GPU RAM, 40GB system RAM.