5 ms·

> OpenAI had decided to put emphasis on larger neural networks fueled by bigger computers and tons of data. So is the answer here both?

• nomel 5y ago
  To make a system noticeably better, it’s often easiest to make all components of the system slightly better.
  • 317070 5y ago
    But that is not what GPT-3 is. You could say that is what they did _given GPT-2_, but that is underselling the complexity. Some scientist at OpenAI had the hypothesis "what if our algorithm is correct, but all we need is to scale it 3 magnitudes larger" and made it happen. They figured out how to scale all the dimensions. How fast should x scale if I scale y? (That is very tricky, as modern machine learning is basically alchemy) And then they actually scaled it. That took a ton of engineering and hardware, for what was essentially still following a hunch. And then they actually noticed how good it was, and did a ton of tests with the surprising results we now all know.
    • nomel 5y ago
      > still following a hunch I'm not sure many would agree that the desire to scale was simply a "hunch".