3 ms·
I'm really curious if/when we'll see MoE based on even smaller models like Phi-2?
by pseudosavant 3y ago
I'm really curious if/when we'll see MoE based on even smaller models like Phi-2?
- lucidyan 3y agoAlready https://nitter.net/maximelabonne/status/1744867841436700850 https://nitter.net/maximelabonne/status/1744867841436700850
- Redster 3y agoAt this small a scale, what would the benefits be of something like a 8x2B as opposed to moving to a 7B?
- pseudosavant 3y agoFor the output performance (quality and tokens/second), Mixtral 8x7B seems to require less VRAM. But it is still hard to make it fit, even with a lot of quantization, within the GPU RAM of most discreet consumer GPUs. Perhaps a smaller base model like Phi-2 could bring the VRAM requirements down, but the MoE will bring the output quality up from Phi-2.