3 ms·
Till now I was using successfully Qwen 3.5 and Gemma 4 at a reasonable speed
by madduci 1mo ago
Till now I was using successfully Qwen 3.5 and Gemma 4 at a reasonable speed
- mobelkh 1mo agowere you running the MoE models? those perform better speed wise
- pyrale 1mo agoThere is no way you would run a dense 27b model on that spec. I ran 3.6 27b on a 64gb ram, 24 gb vram, and it felt like the lower limit for this model with a decent context window. If you want a better experience, maybe wait for either a moe model (like 3.6 35b A3) or a model with less parameters (like 9b). Qwen has been releasing those in the past, so maybe we’ll have them for 3.8 too.
- DanielHB 1mo agoFrom my experience if it doesn't fit on vram it is rarely worth to bother except for a few narrow tasks. For example make an essay about something where you don't actively engage with the LLM after the initial prompt. So mostly one-shot prompts.
- madduci 27d agoThe issue was caused by Ollama. I've tried again with llama.cpp and mounting the igpu device correctly, I get 18-23 tk/s on the Iris Xe card under Debian, MUCH better than before.
- kzrdude 1mo agoI think (maybe I missed something) that identical size and quant versions of Qwen 3.5 and 3.8 should run at the same speed. It’s the exact same architecture.
- madduci 1mo agoTried the 4 Bit versions. It loads, bit the <thinking> output isn't even coming out.