4 ms·
GPT-4 was rumored to be 1.8T params...not 1.2 And the successor model was called "Orion", not "Omni".
by qeternity 2y ago
GPT-4 was rumored to be 1.8T params...not 1.2
And the successor model was called "Orion", not "Omni".
- glenstein 2y agoAppreciate the corrections, but I'm still a bit puzzled. Are they wrong about 4.5 having 12 trillion parameters, it originally intending to be Orion (not omni), or an expected successor to GPT 4? And do you have any related reading that speaks to any of this?
- az226 2y agoGPT-4 was 1.3T. 221B active. 2 experts active. 16 experts total.
- qeternity 2y agoThat sounds correct except for that total parameter count. 110B per expert at 16 experts puts you just shy of 1.8T. Are you suggesting there are ca. 30B shared params between experts?