4 ms·
I can run Qwen3.5-27B-Q4_K_M on my weird PC with 32 GB of system memory and 6 GB of VRAM. It's just a bit slow, is all. I get around 1.7 tokens per second. IMO,
by Wowfunhappy 6mo ago
I can run Qwen3.5-27B-Q4_K_M on my weird PC with 32 GB of system memory and 6 GB of VRAM. It's just a bit slow, is all. I get around 1.7 tokens per second. IMO, everyone in this space is too impatient.
(Intel Core i7 4790K @ 4 Ghz, nVidia GTX Titan Black, 32 GB 2400 MHz DDR3 memory)
Edit: Just tested the new Qwen3.6-27B-Q5_K_M. Got 1.4 tokens per second on "Create an SVG of a pellican riding a bicycle." https://gist.github.com/Wowfunhappy/53a7fd64a855da492f65b4ca73c8fe0d https://gist.github.com/Wowfunhappy/53a7fd64a855da492f65b4ca...
- iamsaitam 6mo agoDon't forget that you're also spending much more electricity because it takes so long to run inference.
- Wowfunhappy 6mo agoGiven current hardware prices I wouldn't expect this to tip the scales. Mind you, for me local models are a fun experiment, for anything serious I would use a frontier model. So I'm not using this for hours a day.
- ranger_danger 6mo agoI have been using Qwen3.5-9B-UD-Q4_K_XL.gguf on an 8GB 3070Ti with llama.cpp server and I get 50-60 tok/s.