5 ms·
More to the argument that we need a model of models - one general one that calls specialists in to do what they are good at and handles that like a foreman for
by hypercube33 3mo ago
More to the argument that we need a model of models - one general one that calls specialists in to do what they are good at and handles that like a foreman for you.
- dofm 3mo agoThis is somewhat akin to the "one-expert-per-query" solution Apple are using in their small foundation models I think?
- kaycey2022 3mo agoIs that different from mixture of experts?
- int_19h 3mo agoYes. A mixture of experts is a single model that activates different routes though the same weights, with the route possibly changing literally on every token. It's not experts as in a bunch of standalone models that are good at specific high-level tasks.
- xnzakg 3mo agoOn the other hand the standalone models would all need "basic" knowledge, so I'm assuming this would be much less disk space-efficient.