3 ms·
GPT-4 was 1.3T. 221B active. 2 experts active. 16 experts total.
by az226 2y ago
GPT-4 was 1.3T. 221B active. 2 experts active. 16 experts total.
- qeternity 2y agoThat sounds correct except for that total parameter count. 110B per expert at 16 experts puts you just shy of 1.8T. Are you suggesting there are ca. 30B shared params between experts?