> It's worth noting that GPT-4 internally uses a Mixture of Experts (MoE) model with 8 'experts' internally, so it's more similar to a multi-agent setup than yo

> It's worth noting that GPT-4 internally uses a Mixture of Experts (MoE) model with 8 'experts' internally, so it's more similar to a multi-agent setup than you might think initially. That's a very common misunderstanding of what MoEs are. What you're describing is an Ensemble. A MoE is where a the input is "routed" to one of X "experts" in the higher layers of the network. Something like this: {input} -> [lower layers] -> <decide which upper layers to send input to> -> [expert] -> output And these experts aren't experts in the space of human defined subjects. Their expertise are in some embedding space, which might or might not line up with our intuition of a subject.