4 ms·
With those GPT4 architecture infos leaked a few weeks ago, my personal theory is that they use mixture-of-experts routing for scaling the system. For instance,
by metanonsense 3y ago
With those GPT4 architecture infos leaked a few weeks ago, my personal theory is that they use mixture-of-experts routing for scaling the system. For instance, they could say: from 0 to 50% system load use a mixture of top-6 models for inference, from 60 to 80% use top-4 models, above 80% use top-2 models. This could make ChatGPT look dumber or smarter depending on the time of the day and system usage. Naturally, the variance of inference quality would spread and people would experience regressions in quality with some probability.
They could also use these parameters over time to become more cost efficient, in general, or reserve GPUs (i.e. allocating more expert networks) for some higher-margin use-cases.
- metanonsense 3y agoThey could even persist the inference parameterization, e.g. "use top-8 for 'compute the sum of 42 and 17'", for every prompt ever seen and reuse it later to prevent people doing "poor man's regression testing" and make the model look more stable.
- ineedasername 3y agoI’m guessing that some tasks relegated to a single one of those experts are tasks that previously benefited from those experts’ capabilities being integrated in a single model. Sort of like having a problem and then asking both a biologist & a chemist for their assessment. But if asked a single biochemist they could more readily provide an answer that synthesized both more comprehensively, even if you got both the biologist and the chemist in the same room with each other.