3 ms·
Consumer-grade hardware? Even at 4bits per param you would need 500GB of GPU VRAM just to load the weights. You also need VRAM for KV cache.
by tintor 11mo ago
Consumer-grade hardware? Even at 4bits per param you would need 500GB of GPU VRAM just to load the weights. You also need VRAM for KV cache.
- CamperBob2 11mo agoIt's MoE-based, so you don't need that much VRAM. Nice if you can get it, of course.