5 ms·

Is the novelty of GPT-3 more about smart algorithms and experienced researchers or is it about beefy hardware/ASICs that do the number crunchings? Or maybe it

Is the novelty of GPT-3 more about smart algorithms and experienced researchers or is it about beefy hardware/ASICs that do the number crunchings? Or maybe it-is-only-about™ gathering and labeling large amount of data for their trainings?

hypothesis 5y ago
> OpenAI had decided to put emphasis on larger neural networks fueled by bigger computers and tons of data. So is the answer here both?
- nomel 5y ago
  To make a system noticeably better, it’s often easiest to make all components of the system slightly better.
  - 317070 5y ago
    But that is not what GPT-3 is. You could say that is what they did _given GPT-2_, but that is underselling the complexity. Some scientist at OpenAI had the hypothesis "what if our algorithm is correct, but all we need is to scale it 3 magnitudes larger" and made it happen. They figured out how to scale all the dimensions. How fast should x scale if I scale y? (That is very tricky, as modern machine learning is basically alchemy) And then they actually scaled it. That took a ton of engineering and hardware, for what was essentially still following a hunch. And then they actually noticed how good it was, and did a ton of tests with the surprising results we now all know.
    - nomel 5y ago
      > still following a hunch I'm not sure many would agree that the desire to scale was simply a "hunch".
317070 5y ago
For GPT-3, they use the data everyone is using. Their cleaning is unique, but so is everyone's. They don't do anything special, but put in some more effort than most though. As far as I can tell, it is a combination of smart algorithms, good engineering, and the hardware to make it happen. And scientists that had the right hunch for which direction to push in.
qeternity 5y ago
> Or maybe it-is-only-about™ gathering and labeling large amount of data for their trainings? Well, GPT-3 isn’t a classifier and it isn’t using labeled data. As an outsider it definitely appears that GPT-3 is an engineering advancement, as opposed to a scientific breakthrough. The difference is important because we need a non linear breakthrough. GPT-3 is a bigger GPT-2. As far as we know, there is no more magic. But I think it’s a near certainty that larger models will not get us to AGI alone.
- ma2rten 5y ago
  As someone in the deep learning community I disagree with your assessment that GPT-3 is a not scientific breakthrough. The GPT-3 paper won a best paper award at the most prestigious machine learning conferences after all. GPT-3 didn't make any modeling advances, but it introduced a completely new paradigm with few-shot learning.
  - mountainriver 5y ago
    Agree, just because it took money and scale to do it doesn’t mean it isn’t a breakthrough.
  - PeterisP 5y ago
    Perhaps we have to distinguish between GPT-3-the-model and GPT-3 paper. IMHO GPT-3 as a model is straighforward engineering, putting a lot of resources in an oversized GPT-2; and while there's significant novelty in the "Language Models are Few-Shot Learners" paper about how exactly you apply these models, that is orthogonal to GPT-3-the-model, the scientific content of that paper applies to any other powerful language models and isn't intimately tied to specifics of GPT-3. In essence, I feel that the same people introduced two quite separate things - a completely new paradigm on how to obtain few-shot learning from a language model in a way that competes with supervised learning of the same tasks; and the GPT-3 large model which is used as "supplementary material" to illustrate that new paradigm bit is also usable and used with the old paradigms, and by itself isn't a breakthrough. And IMHO when the public talks about GPT-3, they do really mean GPT-3-the-model and not the particular few-shot learning approach.
thesausageking 5y ago
It's mainly about their scale which is enabled by their hardware budget (355 GPU years to train it) and their budget for data acquisition / cleaning. GPT-3 has 175B parameters. The previous largest model was Microsoft's Turing-NLG which had 17B. GPT-2 had 1.5B.

hypothesis 5y ago

> OpenAI had decided to put emphasis on larger neural networks fueled by bigger computers and tons of data. So is the answer here both?

nomel 5y ago
To make a system noticeably better, it’s often easiest to make all components of the system slightly better.
- 317070 5y ago
  But that is not what GPT-3 is. You could say that is what they did _given GPT-2_, but that is underselling the complexity. Some scientist at OpenAI had the hypothesis "what if our algorithm is correct, but all we need is to scale it 3 magnitudes larger" and made it happen. They figured out how to scale all the dimensions. How fast should x scale if I scale y? (That is very tricky, as modern machine learning is basically alchemy) And then they actually scaled it. That took a ton of engineering and hardware, for what was essentially still following a hunch. And then they actually noticed how good it was, and did a ton of tests with the surprising results we now all know.
  - nomel 5y ago
    > still following a hunch I'm not sure many would agree that the desire to scale was simply a "hunch".

317070 5y ago

For GPT-3, they use the data everyone is using. Their cleaning is unique, but so is everyone's. They don't do anything special, but put in some more effort than most though. As far as I can tell, it is a combination of smart algorithms, good engineering, and the hardware to make it happen. And scientists that had the right hunch for which direction to push in.

qeternity 5y ago

> Or maybe it-is-only-about™ gathering and labeling large amount of data for their trainings? Well, GPT-3 isn’t a classifier and it isn’t using labeled data. As an outsider it definitely appears that GPT-3 is an engineering advancement, as opposed to a scientific breakthrough. The difference is important because we need a non linear breakthrough. GPT-3 is a bigger GPT-2. As far as we know, there is no more magic. But I think it’s a near certainty that larger models will not get us to AGI alone.

ma2rten 5y ago
As someone in the deep learning community I disagree with your assessment that GPT-3 is a not scientific breakthrough. The GPT-3 paper won a best paper award at the most prestigious machine learning conferences after all. GPT-3 didn't make any modeling advances, but it introduced a completely new paradigm with few-shot learning.
- mountainriver 5y ago
  Agree, just because it took money and scale to do it doesn’t mean it isn’t a breakthrough.
- PeterisP 5y ago
  Perhaps we have to distinguish between GPT-3-the-model and GPT-3 paper. IMHO GPT-3 as a model is straighforward engineering, putting a lot of resources in an oversized GPT-2; and while there's significant novelty in the "Language Models are Few-Shot Learners" paper about how exactly you apply these models, that is orthogonal to GPT-3-the-model, the scientific content of that paper applies to any other powerful language models and isn't intimately tied to specifics of GPT-3. In essence, I feel that the same people introduced two quite separate things - a completely new paradigm on how to obtain few-shot learning from a language model in a way that competes with supervised learning of the same tasks; and the GPT-3 large model which is used as "supplementary material" to illustrate that new paradigm bit is also usable and used with the old paradigms, and by itself isn't a breakthrough. And IMHO when the public talks about GPT-3, they do really mean GPT-3-the-model and not the particular few-shot learning approach.

thesausageking 5y ago

It's mainly about their scale which is enabled by their hardware budget (355 GPU years to train it) and their budget for data acquisition / cleaning. GPT-3 has 175B parameters. The previous largest model was Microsoft's Turing-NLG which had 17B. GPT-2 had 1.5B.