2 ms·
More like `ollama launch claude --model qwen3.6:latest` Also you need to check your context size, Ollama default to 4K if <24 Gb of VRAM and you need 64K minim
by Ladioss 6mo ago
More like `ollama launch claude --model qwen3.6:latest`
Also you need to check your context size, Ollama default to 4K if <24 Gb of VRAM and you need 64K minimum if you want claude to be able to at least lift a finger.
- txtsd 6mo agoI only have 16GB VRAM, and my system uses ~4GB from that. What are my options? I got this one: `Qwen3.6-35B-A3B-UD-IQ2_XXS.gguf`
- Ladioss 6mo agoMy system has 16 Gb VRAM / 32 Gb RAM, and ollama runs qwen3.6:latest at decent speed just fine. The 35b model is a moe, so I guess the whole model is offloaded.
- Patrick_Devine 6mo agoIf you're on a Mac, use the MLX backend versions which are considerably faster than the GGML based versions (including llama.cpp) and you don't need to fiddle with the context size. The models are `qwen3.6:35b-a3b-nvfp4`, `qwen3.6:35b-a3b-mxfp8`, and `qwen3.6:35b-a3b-mlx-bf16`.
- egorfine 6mo agoI was comparing various models at M5 Pro 48GB RAM MLX vs GGUF and found that MLX models have a higher time to first token (sometimes by an order of magnitude) while tokens/sec and memory usage is same as GGUF. Gemma 3 27B q4: * MLX: 16.7 t/s, 1220ms ttft * GGUF: 16.4 t/s, 760ms ttft Gemma 4 31B q8: * MLX: 8.3 t/s, 25000ms ttft * GGUF: 8.4 t/s, 1140ms ttft Gemma 4 A4B q8: * MLX: 52 t/s, 1790ms ttft * GGUF: 51 t/s, 380ms ttft All comparisons done in LM Studio, all versions of everything are the latest.