3 ms·
For qwen3.6-27b you can also run the q4 variant with full ~250K context on one 3090. It's fast enough to not be frustrating so the speed gains with 2x 3090s wou
by turova 3mo ago
For qwen3.6-27b you can also run the q4 variant with full ~250K context on one 3090. It's fast enough to not be frustrating so the speed gains with 2x 3090s wouldn't be worth it to me. Running a q6 on 2x 3090s at half the speed with a smaller context is an option, but you're really not going to compete with SOTA models there anyway so unless you already have 2x 3090s, I would say 1 is the best investment given current prices. It's good enough to do a lot, especially with a well-configured harness.
- hypfer 3mo agoThat math (250k context, Q4 model, 24GB VRAM) only checks out at q4 quant for the K/V cache, which is probably not the best idea.
- nabakin 3mo agoAre you running qwen3.6-27b on one 3090 with your KV cache at q4? Ime there is significant long-context recall accuracy degradation at that precision. I prefer putting the KV cache at q8 and working with the 120k context
- Der_Einzige 3mo agoUse modern samplers and you don’t need to limit yourself to 8bit at half the context window. I could push it down to 1.58 bits and get decently good output easily by simply not using the garbage default top_p and top_k that vendors continue to wrongly recommend.
- anon373839 3mo agoWhere do you find optimal samplers and sampler settings for these models? Very interested in this as I, too, use Q8.