3 ms·
Several people hinted/remarked (starting w/ George Hotz, then others more closely linked to OpenAI) that it's a Mixture of Experts* approach comprised of 8 220B
by brandall10 3y ago
Several people hinted/remarked (starting w/ George Hotz, then others more closely linked to OpenAI) that it's a Mixture of Experts* approach comprised of 8 220B parameter models.
* https://arxiv.org/pdf/2101.03961.pdf https://arxiv.org/pdf/2101.03961.pdf
- j45 3y agoI wonder if additional layers of factories of factories approach can continue to improve it. I'm not familiar enough with the technology but could it be possible to create a prompt, or multiple prompts to stitch together 8 similtaneous calls to GPT3.5 pulled together and see if the quality is similar to GPT4?
- andai 3y agoI think the experts part implies that each "GPT-3" has been specialized in some way?
- brandall10 3y agoThey're not 3.5 models, which are 175B param, they're 220B and the idea is the fine tuning is different for each which is where the 'expert' comes in.
- j45 3y agoAh, that makes sense, thanks for clarifying