3 ms·
Not a SWE, but I do Freetoken (backend) + unsloth (front) to run qwen 3.6 35B A3B NVFP4 (~20GB) in 8GB VRAM + ~20GB RAM. Getting around 50 t/s on a 3070. I don'
by gentile 9d ago
Not a SWE, but I do Freetoken (backend) + unsloth (front) to run qwen 3.6 35B A3B NVFP4 (~20GB) in 8GB VRAM + ~20GB RAM. Getting around 50 t/s on a 3070. I don't think I can even share through this site.
- scoring1774 9d agoIf not coding what do you use it for? Also what kind of performance do you tend to get out of it? I've been thinking about a similar setup but am skeptical it's worth the effort.
- gentile 9d agoIt's still typically for code, like python data stuff, bash scripts, general web search, small javascript stuff for my website. By saying not SWE, I mean I don't really see much benefit from "agentic" stuff, although I've tried. 50 t/s means 40~60 seconds for a typical thinking response. I used to run gemma 4 e4b-it-qat fully in GPU (~150 t/s), but the quality improvement moving to a much larger MoE model was 100% worth the switch. Especially because I had a lot of idle ram (from the before times :( ). There is some tradeoff in terms of context length, but I just use it as a chat. Also it's fairly trivial to setup as long as the GPU is supported.