2 ms·
Qwen2.5 has a 32B release, and quantised at q5_k_m it *just about" completely fills a 4090. It's a good model, too.
by regularfry 2y ago
Qwen2.5 has a 32B release, and quantised at q5_k_m it *just about" completely fills a 4090.
It's a good model, too.
- kristianp 2y agoDo you also need space for context on the card to get decent speed though?
- regularfry 2y agoDepends how much you need. Dropping to q4_k_m gives you 3GB back if that makes the difference.