3 ms·
You can run the 27B released last week. I haven't tried it yet myself but the 3.6 version runs great on my 5090.
by cpburns2009 2mo ago
You can run the 27B released last week. I haven't tried it yet myself but the 3.6 version runs great on my 5090.
- Philpax 2mo agoStrongly recommend https://github.com/Neroued/ninfer https://github.com/Neroued/ninfer, which can pull ~180 TPS on 5090 with 3.8, and 500 (!) with 3.6 35B-A3B.
- cpburns2009 2mo agoI've been waiting for the dust to settle on this model so I can find a good runtime setup. I'm definitely bookmarking this. Thanks!
- hasteg 2mo agoI've been running 27B a lot, I am honestly shocked at how well it performs. It's mind blowing how well the small qwen models (and particular, the 3.8 model) runs locally. It can create some really impressive toy coding projects. I haven't really done too much integrating into my actual workflows because I pay $100 for Claude Max, but I can see it being pretty helpful in that.