3 ms·
4090 has 24G So we really need ~40B or G model (two cards) or like a ~20B with some room for context window. 5090 has ??G - still unreleased
by foxhop 2y ago
4090 has 24G
So we really need ~40B or G model (two cards) or like a ~20B with some room for context window.
5090 has ??G - still unreleased
- deleted 2y ago[deleted]
- regularfry 2y agoQwen2.5 has a 32B release, and quantised at q5_k_m it *just about" completely fills a 4090. It's a good model, too.
- kristianp 2y agoDo you also need space for context on the card to get decent speed though?
- regularfry 2y agoDepends how much you need. Dropping to q4_k_m gives you 3GB back if that makes the difference.