3 ms·
How many parameters did ChatGPT have in Dec 2022 when it first broke into mainstream news?
by agnishom 2y ago
How many parameters did ChatGPT have in Dec 2022 when it first broke into mainstream news?
- simonw 2y agoI don't think that's ever been shared, but it's predecessor GPT-3 Da Vinci was 175B. One of the most exciting trends of the past year has been models getting dramatically smaller while maintaining similar levels of capability.
- reissbaker 2y agoGPT-3 had 175B, and the original ChatGPT was probably just a GPT-3 finetune (although they called it gpt-3.5, so it could have been different). However, it was severely undertrained. Llama-3.1-8B is better in most ways than the original ChatGPT; a well-trained ~70B usually feels GPT-4-level. The latest Llama release, llama-3.3-70b, goes toe-to-toe even with much larger models (albeit is bad at coding, like all Llama models so far; it's not inherent to the size, since Qwen is good, so I'm hoping the Llama 4 series is trained on more coding tokens).
- swyx 2y ago> However, it was severely undertrained by modern standards. at the time, it was trained according to neural scaling laws oai believed to hold.
- reissbaker 2y agoSure, at the time everyone misunderstood Chinchilla. Nonetheless it was severely undertrained, even if they didn't know it back then.