3 ms·
Interesting implications for future optimization. If you don't have enough RAM to fine-tune the whole model, you could use this to select the two most relevant
by sp332 3y ago
Interesting implications for future optimization. If you don't have enough RAM to fine-tune the whole model, you could use this to select the two most relevant experts for each layer of the model. Maybe for inference, you could quantize the less-relevant experts to be smaller.
- make3 3y agoI would assume that you would get a big performance drop, because knowledge is split in a very non human understandable way between the experts
- Tostino 3y agoSo use a non human algorithm to optimize that problem by maybe collecting some huristics during a benchmark phase, and then the quantization step can take that into account based on specific pathways that are never hit because another expert is always better at that task and always gets picked. It seems like there is a ton of room for improvement there.
- ajtejankar 3y agoFixing the experts for a layer might not work since all experts fire almost with equal probability. There are small variations by topic but they are consistent enough to be captured with a simple linear classifier. I believe this happens due to the load balancer loss which forces the model to pick all experts with equal probability. However, what you're saying is a great direction for future MoE's. Can we train MoEs without load balancing so that it is possible to quantize/prune the non-relevant experts more aggressively? We haven't had any major open source MoE's because, as far as I know, they are not straightforward to train, but I expect this to change.
- sp332 3y agoI could be reading this wrong, but doesn't the article say that the MMLU topic is enough to determine which experts got picked 96% of the time? Edit: Oh I read it completely backwards, didn't I. Given the decisions, you can determine the topic of the input.