4 ms·
I am curious if there has been work to remove experts from an open-weights model. The goal would be to reduce the size to be able to run on desktop GPUs without
by bwfan123 2mo ago
I am curious if there has been work to remove experts from an open-weights model. The goal would be to reduce the size to be able to run on desktop GPUs without compromising quality. For a focused usecase - say coding, you dont need a model that knows world history. And, I am not talking about quantization. If it is possible to determine which experts are active for some usecases, and surgically remove the others.
- smallerize 2mo agoExperts aren't trained on separate tasks. More recent routers are designed to spread out requests even more evenly, and they were already pretty even.
- monster_truck 2mo agoGlazing over a lot, that's how they work already, just not in the way you think. A relatively small fraction of the model is active at any given time
- Tepix 2mo agoYes. It's called REAP and from what I've seen, results aren't stellar.