3 ms·
I've been largely using Qwen3.5-122b at 6 bit quant locally for some c++/go/python dev lately because it is quite capable as long as I can give it pretty specif
by jwitthuhn 6mo ago
I've been largely using Qwen3.5-122b at 6 bit quant locally for some c++/go/python dev lately because it is quite capable as long as I can give it pretty specific asks within the codebase and it will produce code that needs minimal massaging to fit into the project.
I do have a $20 claude sub I can fall back to for anything qwen struggles with, but with 3.5 I have been very pleased with the results.
- 3836293648 6mo agoHow much VRAM do you need for that?
- seemaze 6mo agoI squeeze Qwen3.5-122B-A10B at Q6 into 128GB. It's a great model.
- mistercheese 6mo agoWow what kind of hardware do you have? Mac Studio, dgx spark, strix halo? How fast is it?
- seemaze 6mo agoStrix Halo, I'm seeing performance inline with these results[0]. I'm interested to investigate the claimed gains from the lemonade-sdk port of Apple MLX inference[1]. [0]https://kyuz0.github.io/amd-strix-halo-toolboxes/ https://kyuz0.github.io/amd-strix-halo-toolboxes/ [1]https://github.com/lemonade-sdk/lemonade/issues/1642 https://github.com/lemonade-sdk/lemonade/issues/1642
- canpan 6mo agoNot OP, but I ran 122b successfully with normal RAM offloading. You dont need all that much VRAM, which is super expensive. I used 96gb ram + 16gb vram gpu. But it's not very fast in that setup, maybe 15 token per second. Still, you can give it a task and come back later and its done. (Disclaimer: I build that PC before stuff got expensive)
- jwitthuhn 6mo ago128GB on a mac with unified memory. The model itself takes something like 110 of that and then I have ~16 left over to hold a reasonably sized context and 2 for the OS. I do have a dedicated machine for it though because I can't run an IDE at the same time as that model.