3 ms·
Another naive question: why is this better than creating a separate, smaller model for each expert?
by stingraycharles 6y ago
Another naive question: why is this better than creating a separate, smaller model for each expert?
- p1esk 6y agoYes, you can think of this as a collection of small models, with another model choosing which smaller model to use for each input.
- gwenzek 6y agoNot really, the dispatching between experts happen for every word and every layer, so you can't easily isolate distinct models.
- probably_wrong 6y agoThe common argument I've heard: because then you would have to decide how many experts models are required, train and evaluate them separately, and overall make your architecture dependent on this choice. If your expert is wrong and miscalculates how many models are required then your entire architecture is also likely to be wrong (humans, am I right?). Researchers at Google's scale prefer a single model where you throw all your data in a single bin and get perfect performance out, no tweaking and no pesky humans required.
- stingraycharles 6y agoBut this is something you could just use a hp search for, right, to determine the amount of models? Or are hp searches generally not used anymore at that scale?
- thomasahle 6y agoHow would you know which inputs to use for training each expert?