3 ms·
How are you getting Qwen3:30B-A3B running with 8GB? On my system it takes 20GB of VRAM to launch it.
by tedivm 1y ago
How are you getting Qwen3:30B-A3B running with 8GB? On my system it takes 20GB of VRAM to launch it.
- fennecfoxy 1y agoProbably offload to regular ram I'd wager. Or really, really, reaaaaaaally quantized to absolute fuck. Qwen3:30B-A3B Q1 with a 1k Q4 context uses 5.84GB of vram.
- kekePower 1y agoIt offloads to system memory, but since there are "only" 3 Billion active parameters, it works surprisingly well. I've been able to run models that are up to 29GB in size, albeit very, very slow on my system with 32GB RAM.