3 ms·
I think this is called "mixture of experts" and also there's a lot of speculation that it's how GPT-4 works, although probably with just a few large models rath
by esperent 3y ago
I think this is called "mixture of experts" and also there's a lot of speculation that it's how GPT-4 works, although probably with just a few large models rather than many small ones.
- jmiskovic 3y agoIt's been confirmed by multiple (unofficial) sources that GPT-4 is 8 models, each 220B parameters. Another rumor is GPT-4 being 16x111B models. There's a quite fresh and active project replicating something similar with herd of llamas: https://github.com/jondurbin/airoboros https://github.com/jondurbin/airoboros
- codercowmoo 3y agoCan you provide some of these sources?
- jmiskovic 3y agoAll of these should have close ties with OpenAI, but no one from inside confirmed it as far as I know. https://twitter.com/swyx/status/1671272883379908608 https://twitter.com/swyx/status/1671272883379908608 https://twitter.com/soumithchintala/status/1671267150101721090 https://twitter.com/soumithchintala/status/16712671501017210... https://twitter.com/MParakhin/status/1670666605427298304 https://twitter.com/MParakhin/status/1670666605427298304