2 ms·
But is that not what MoE is supposed to solve already? Why no it bake whatever Cosine does into the coding expert?
by jstummbillig 2y ago
But is that not what MoE is supposed to solve already? Why no it bake whatever Cosine does into the coding expert?
- everforward 2y agoI am not an expert or even particularly knowledgeable about this, so treat the following as “armchair speculation by an enthusiast”. Wikipedia says that as of 2023, each expert is ~10 billion parameters, so they’re still much smaller than something like Deepseek Coder 70B. I’m not sure why they’re so small though. I don’t know whether there’s some kind of architectural issue or super linear scaling somewhere preventing them from growing, or if they’re just trying to keep inference costs down.