2 ms·
Even when quantized down to 4 bits to fit on a 4090?
by ls612 1y ago
Even when quantized down to 4 bits to fit on a 4090?
- FuckButtons 1y agoNot in my experience, running qwen3:32b is good, but it’s not as coherent or useful as 3.5 at a 4bit quant. But the gap is a lot narrower than llama 70b.