You shouldn't take the "mixture of experts" too literally, it's yet another architecture to use internally for a gradient descent optimized graph of ops. I obv

You shouldn't take the "mixture of experts" too literally, it's yet another architecture to use internally for a gradient descent optimized graph of ops. I obviously don't know how GPT-4 do it (or if it even does it) but think of partitioning your network into a couple of very isolated sub-graphs (the "experts"), and add another learnable network between the input tokens and the experts, that learns to route tokens to 1 or more expert sub-graphs. Then the gain is that you can potentially ignore running the unused sub-graphs completely for that token, and you can distribute them on other GPUs as except for the input and output they are independent of each other. It all depends on the problem, data, and if the gradient descent optimizer can find a way to actually partition the problem usefully using the router and "experts".