3 ms·
> I still wonder what makes GPT-4 so much better than its contemporaries. OpenAI have had many years to craft their dataset down from the noisy public datasets
by losteric 3y ago
> I still wonder what makes GPT-4 so much better than its contemporaries.
OpenAI have had many years to craft their dataset down from the noisy public datasets, and GPT4 is (supposedly) a mixture of 8 "expert models" each of which is 220B (5x+ larger than the Falcon 40B) with a total of 1.7B parameters (3x+ Google's huge 540B PaLM). The hardware and software to train networks of that scale is also a deep moat. Relatively speaking, the model architecture ("gpt from scratch") is the easiest piece.