But that is not what GPT-3 is. You could say that is what they did _given GPT-2_, but that is underselling the complexity. Some scientist at OpenAI had the hyp

But that is not what GPT-3 is. You could say that is what they did _given GPT-2_, but that is underselling the complexity. Some scientist at OpenAI had the hypothesis "what if our algorithm is correct, but all we need is to scale it 3 magnitudes larger" and made it happen. They figured out how to scale all the dimensions. How fast should x scale if I scale y? (That is very tricky, as modern machine learning is basically alchemy) And then they actually scaled it. That took a ton of engineering and hardware, for what was essentially still following a hunch. And then they actually noticed how good it was, and did a ton of tests with the surprising results we now all know.