3 ms·
Are 5090's able to run 32B models?
by osti 2y ago
Are 5090's able to run 32B models?
- regularfry 2y agoThe 4090 can run 32B models in Q4_K_M, so yes, on that measure. Not unquantised though, nothing bigger than Q8 would fit. On a 32GB card you'll have more choices to trade off quantisation against context.