3 ms·
Large MoE models are more socially accepted because medium/large sized MoE models can still be quite small wrt. expert size (which is what sets the amount of re
by zozbot234 10mo ago
Large MoE models are more socially accepted because medium/large sized MoE models can still be quite small wrt. expert size (which is what sets the amount of required VRAM). But a large dense model is still challenging to get to run.
- coder543 10mo agoI meant large MoE models are more socially accepted now. They were not when Llama 4 launched, and I believe that worked against the Llama 4 models. The Llama 4 models are MoE models, in case you are unaware, since it feels like your comment feels was implying they were dense models.