2 ms·
It's fine for dense models where you need them in VRAM, less so for MoE where you're offloading layers to ram. But 32/32 is pretty good for both in the popular
by tempoponet 4mo ago
It's fine for dense models where you need them in VRAM, less so for MoE where you're offloading layers to ram. But 32/32 is pretty good for both in the popular ~30b range right now.
- xxs 4mo agorunning 5090 on 32GB RAM is just weird, still