24 ms·
GPT-3 has 175 billion parameters. So they need to scale by 64x. They already have a comparable amount of data than what was used by OpenAI, so it's about scalin
by Voloskaya 6y ago
GPT-3 has 175 billion parameters. So they need to scale by 64x. They already have a comparable amount of data than what was used by OpenAI, so it's about scaling the numbers of GPUs.
- f430 6y agoI see so that means this GPT Neo is 64 less powerful?
- Voloskaya 6y agoAccuracy and numbers of parameters don't scale linearly together. It varies widely depending on exactly what you are measuring accuracy on as well etc. But a very approximate rule of thumb would be to say that accuracy scales with the log of the parameter count (for the same architecture).