3 ms·
At this small a scale, what would the benefits be of something like a 8x2B as opposed to moving to a 7B?
by Redster 3y ago
At this small a scale, what would the benefits be of something like a 8x2B as opposed to moving to a 7B?
- pseudosavant 3y agoFor the output performance (quality and tokens/second), Mixtral 8x7B seems to require less VRAM. But it is still hard to make it fit, even with a lot of quantization, within the GPU RAM of most discreet consumer GPUs. Perhaps a smaller base model like Phi-2 could bring the VRAM requirements down, but the MoE will bring the output quality up from Phi-2.