3 ms·
It's been confirmed by multiple (unofficial) sources that GPT-4 is 8 models, each 220B parameters. Another rumor is GPT-4 being 16x111B models. There's a quite
by jmiskovic 3y ago
It's been confirmed by multiple (unofficial) sources that GPT-4 is 8 models, each 220B parameters. Another rumor is GPT-4 being 16x111B models.
There's a quite fresh and active project replicating something similar with herd of llamas: https://github.com/jondurbin/airoboros https://github.com/jondurbin/airoboros
- codercowmoo 3y agoCan you provide some of these sources?
- jmiskovic 3y agoAll of these should have close ties with OpenAI, but no one from inside confirmed it as far as I know. https://twitter.com/swyx/status/1671272883379908608 https://twitter.com/swyx/status/1671272883379908608 https://twitter.com/soumithchintala/status/1671267150101721090 https://twitter.com/soumithchintala/status/16712671501017210... https://twitter.com/MParakhin/status/1670666605427298304 https://twitter.com/MParakhin/status/1670666605427298304