This isn't how neural networks work. For vanilla models, you always use all the weights. That isn't true for mixture-of-experts, though, and in that setting, y

This isn't how neural networks work. For vanilla models, you always use all the weights. That isn't true for mixture-of-experts, though, and in that setting, your approach has merit.