4 ms·
Does anyone know what Figure 8 at the end shows? It looks like each expert is used interchangeably with no clear pattern. And earlier they say "Surprisingly, w
by invert_franklin 3y ago
Does anyone know what Figure 8 at the end shows?
It looks like each expert is used interchangeably with no clear pattern. And earlier they say "Surprisingly, we do not observe obvious patterns in the assignment of experts based on the topic."
So then, what is the point of the "expert"?
Could this extra performance just be through the 8 expert architectural design, and not be based on the underlying training material? For example, if all 8 experts were ArXiv papers, would the performance be different?
- jakderrida 3y agoNot sure if it answers your question, but the expertise of each expert arise from the training process and aren't assigned by a human. Thus, they wouldn't be discernable to a human. The choice of the model to use, I think, is something called like a "gating network". That's also trained to favor most appropriate model based on training.