4 ms·
Isn't this what a MoE LLM does already?
by 3abiton 2y ago
Isn't this what a MoE LLM does already?
- sodality2 2y agoMoE-based models are one model with multiple experts. This solution could use entirely different models with different architectures (and probably supports MoE models itself)
- danlenton 2y agoexactly!
- danlenton 2y agoMoE LLMs use several "expert" fully connected layers, which are routed to during the forward pass, all trained end-to-end. This approach can also work with black-box LLMs like Opus, GPT4 etc. It's a similar concept but operating at a higher level of abstraction.