4 ms·
It's mainly about their scale which is enabled by their hardware budget (355 GPU years to train it) and their budget for data acquisition / cleaning. GPT-3 has
by thesausageking 5y ago
It's mainly about their scale which is enabled by their hardware budget (355 GPU years to train it) and their budget for data acquisition / cleaning.
GPT-3 has 175B parameters. The previous largest model was Microsoft's Turing-NLG which had 17B. GPT-2 had 1.5B.