The rumor here is that GPT-4 isn't one giant model, but instead a "mixture of experts" combination of 8 smaller GPT-3 sized models with different training sets.

The rumor here is that GPT-4 isn't one giant model, but instead a "mixture of experts" combination of 8 smaller GPT-3 sized models with different training sets. This has implications for what may be possible on consumer hardware.