3 ms·
180b * 1.76 bits per weight = 39.6 gigabytes. Best you could realistically run in 32gb is like 28gb, or a 127B param model
by kennywinker 17d ago
180b * 1.76 bits per weight = 39.6 gigabytes.
Best you could realistically run in 32gb is like 28gb, or a 127B param model
- jokethrowaway 16d agoQwen3.8-Next, thanks to its new architecture, is quite fast even if part of it is streaming from disk
- kennywinker 16d agoTotally. Any MoE model can have experts swapped in and out from disk or system ram. I only framed it this way because the question was about the model fitting in vram.