5 ms·

All valid points, but I disagree with the conclusion, for several reasons: * First of all, the GPT-3 authors successfully trained a model with 175 billion para

by cs702 6y ago

All valid points, but I disagree with the conclusion, for several reasons: * First of all, the GPT-3 authors successfully trained a model with 175 billion parameters. I mean, 175 billion. The previous largest model in the literature, Google’s T5, had "only" 11 billion. Models with trillions of weights are suddenly looking... achievable. That's a significant experimental accomplishment. * Second, the model achieves competitive results on many NLP tasks and benchmarks without finetuning, using only a context window of text for instructions and input. There is only unsupervised (i.e., autoregressive) pretraining. AFAIK, this is the first paper that has reported a model doing this. It's a significant experimental accomplishment that points to a future in which general-purpose NLP models could be used for novel tasks without requiring additional training from the get-go. * Finally, the model’s text generation fools human beings without having to cherry-pick examples. AFAIK, this is the first paper that has reported a model doing this. It's another significant experimental accomplishment. More generally, I find that some AI researchers and practitioners with strong theoretical backgrounds tend to dismiss this kind of paper as "merely" engineering. I think this tendency is misguided. We must build giant machines and gather experimental evidence from them -- akin to physicists who build giant high-energy particle colliders to gather experimental evidence from them. I'm reminded of Rich Sutton's essay, "The Bitter Lesson:" http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html

100721 6y ago
I mostly agree with you. However, > It's a significant experimental accomplishment that points to a future in which general-purpose NLP models could be used for novel tasks without requiring additional training from the get-go. This premise is still purely science fiction. This model does not touch on either novel tasks nor being free from pretraining (unless I misunderstand). But overall, I think you’re right: it’s significant for a number of reasons.
- cs702 6y ago
  GPT-3 was pretrained on five datasets (Common Crawl, WebText2, Books1, Books2, and Wikipedia; see table 2.2), and then used on previously unseen tasks (Q&A, translation, cloze, etc.) without finetuning, i.e., weights were not updated after the original (autoregressive) pretraining. This promises a possible future in which general-purpose models are pretrained once, and deployed to production for multiple tasks.
  - pacala 6y ago
    This is pretty mind-boggling, to the point of suspecting an error in the methodology. If it is just a language model, then it has no baked in notion of test time tasks. How on earth does a language model know what is required of it at test-time without fine tuning? How does it know that the test time prompt are examples of the task, and not some story prompt it's supposed to riff off in random ways?
    - cs702 6y ago
      Yes, mind-boggling. For each task, the authors feed a context window of text with either zero to a few sample queries and responses, followed by a query without the response. The model generates a response for the last query. BTW, this approach is analogous to what you would do with a human being: you would provide zero to a few sample questions and answers, and then ask a question.
      - deleted 6y ago
        [deleted]
chongli 6y ago
I'm reminded of Rich Sutton's essay, "The Bitter Lesson:" Moore's law is running on fumes at this point. The complexity of further scaling has reached geopolitical proportions. We need to get back to looking at more creative models in both the software and hardware domains. the model achieves competitive results on many NLP tasks and benchmarks without finetuning The article dismisses this result with the following analogy: “No, my 10-year-old math prodigy hasn’t proven any new theorems, but she can get a perfect score on the math SAT in under 10 minutes. Isn’t that groundbreaking?” And I tend to agree. We've had a game of benchmarking brinkmanship for a while now. At what point are we going to see some groundbreaking applications?
- cs702 6y ago
  I never said otherwise. Like physicists, we need both new theory (for new insights, ideas, models, etc.) and new experiments (for replication, performance, scalability, etc.) Also, Moore's Law is running on fumes, yes, but there's quite a bit of R&D focused on coming up with hardware that massively scales up (e.g., by more efficiently parallelizing) the dense and sparse multiply-sum operations common to so many AI models. I think Sutton's point about models that leverage computation is spot-on.
- tal8d 6y ago
  > We need to get back to looking at more creative models in both the software and hardware domains. That would be pretty foolish, given the fact that every hand crafted model eventually gets surpassed with brute force. A better use of time would be tackling whatever you mean by "complexity of further scaling has reached geopolitical proportions". I'm not a fan of it, as it is terribly inelegant, but denying the years of consistent brute force wins would just be silly. The best strategy for any nation with an interest in AI (be it economic or something much more skynety) would be securing two things very quickly: fabrication capacity and nuclear power, because this stuff is going to be measure megawatts - not ANN layers. Improving the efficiency of that conversion would certainly be helpful, but history has shown that to be a lower priority, just take a look at how ridiculously deep software stacks are compared to 20 years ago. I really wish the linguists had been proven right in the 1970s...
rmrfstar 6y ago
> practitioners with strong theoretical backgrounds tend to dismiss this kind of paper as "merely" engineering This paper implements an architecture that will be out of reach for me for about 5 years. So, I ask myself "why will this paper matter in 5 years?" There are two reasons I can imagine: 1. It shows that there is no phase change in the size-performance trend already documented over many orders of magnitude. 2. It was used as input data for pruning or distillation algorithms rooted in a better understanding of why language models work. If the NLP community remains laser focused on hill-climbing compute-agnostic benchmarks, I don't think there will be enough people working on 2. If 1 is the only reason it matters, I struggle to see how it is worth the cost in high-end research talent. It feels like standard low-risk corporate iteration.
- cs702 6y ago
  > "Why will this paper matter in 5 years?" 3. First evidence that performance continues to improve at hundreds of billions of weights -- paving the way for trillions of weights, approaching orders of magnitude comparable to that of the human brain connectome. 4. First evidence (AFAIK) that larger NLP models do not need task-specific finetuning -- paving the way for general-purpose models that work well on any NLP task without additional training. 5. First evidence (AFAIK) that larger NLP models fool human beings without cherry-picking -- paving the way for models that can pass ever more challenging Turing tests.
  - deleted 6y ago
    [deleted]
- jeeceebees 6y ago
  I think the larger models get, the more incentive there is for researchers to look into pruning/distilling them for practical use. GPT-1,2,3 et al. have all shown that larger is better. While in the short term this means people will simply throw larger and larger clusters at the problem, in the longer term there needs to be inovation in making it more efficient on the clusters we have (as even the cloud has limits). I think sheer parameter count is an important part of the equation in general intelligence, so it's important that there are labs that work on scaling up promising leads to trillions of parameters on top of labs thinking of new promising directions.
Eridrus 6y ago
I'm an engineer and this is a boring paper. They didn't beat the conclusively beat SOTA the way BERT/T5 beat basically everything. Sure, this is "unsupervised", but from an engineering perspective that's entirely uninteresting, so we don't even get actual scaling numbers. This paper was written to pretend it is more than it is, which is basically the hallmark of everything OpenAI does.
- The_rationalist 6y ago
  There is one model that consistently outperform BERT: it's XLnet but it's already one year old, innovation seems to slow down
Al-Khwarizmi 6y ago
> More generally, I find that some AI researchers and practitioners with strong theoretical backgrounds tend to dismiss this kind of paper as "merely" engineering. It's just that this kind of work is more interesting as a general member of the public than as an AI researcher. As a human being I find it really interesting to see where this kind of models can take us. I was amazed playing with GPT-2 online demos and seeing to what extent it could generate text that looked like what a human could produce. With its quirks and problems, but still impressive. And I can't wait to put my fingers on a GPT-3 online demo. But as an NLP academic researcher (and this is not hypothetical, I'm actually one), what do I learn from this paper? What importance does it have to my research? Actually very little. You need more than 350 GB to fit the 175B parameters in memory, currently the largest GPU I can access has 24 GB (and I can access only one of those, which I use to -barely- run BERT-large). The cost of training the model in the cloud is estimated to be $12 million (https://twitter.com/eturner303/status/1266264358771757057 https://twitter.com/eturner303/status/1266264358771757057). This is a single training run, not including any neural architecture search, bug fixing, etc. So even though for an academic researcher my funding situation is not bad at all, I'm like a couple of orders of magnitude away from being able to do anything meaningful with models of this size, and can't expect that to change for at least 8-10 years (by which point, at the pace NLP evolves, this will be ancient history anyway). On the other hand, of course very often you learn useful ideas from papers that you can apply yourself even if it's not by implementing the same models in the paper, but that's not the case either. Here the lesson learned is "bigger is better" and I cannot train these enormous models, so there is not really much here that I can apply. So as an academic researcher, really there isn't a lot to do with this apart from shrugging, and basically dismissing it and just keeping trying to do our best with what we have. Which is still useful, at least if we don't want NLP applications to be in the hands of an oligopoly of megacorps and restricted only to the few most economically viable languages.
- andi999 6y ago
  I am doing other hpc stuff, so I am wondering why are you limited to 1 GPU? Windows recognizes up to 8 I think.
  - 6y ago
KKKKkkkk1 6y ago
I don't know why this reminds you of Sutton's essay. Sutton claims that algorithmic work will be superseded by brute-force approaches powered by exponential growth in computing power. But this paper is not the result of exponential growth in computing power. It's the result of $1b worth of Azure credits provided for free by Microsoft. Rich Sutton is a great scientist but he is fooled by randomness. He initiated his research program just as Moore's law was taking off. Thanks to Moore, his approach saw incredible success and brought him deserved acclaim. But just as Moore's law is pulling the rug from under him, he is using his stature to claim that no other approach but his can work.
- justicezyx 6y ago
  The engineering effort of utilizing 1B worth of Azure credits is surely worth at least 10% of it.
  - master_yoda_1 6y ago
    Yes then credit to azure engineers not open ai people
    - justicezyx 6y ago
      Wow, do you reasonably believe OpenAI people are not contributing?... I thought HN visitors are not capable of that level of naive thinking.
      - deleted 6y ago
        [deleted]
- clickok 6y ago
  You have misunderstood Sutton's argument. Quoting Rich: One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The point isn't that improvements in our algorithms is unnecessary or unhelpful, rather that the algorithms we should focus on will be capable of scaling with arbitrary amounts of compute/data. Such as, for example, neural networks, where we see an almost constant rate of improvement (for the appropriate architecture) as more resources are added.
gentleman11 6y ago
Think about the waves of fake news to come. Soon every corporation (and political group) in American will have a data center dedicated to overloading social media with messaging that benefits them. At least right now it’s not so hard to spot the bots. I am having some trouble seeing the benefits in lieu of the mess that is coming.
papeda 6y ago
> More generally, I find that some AI researchers and practitioners with strong theoretical backgrounds tend to dismiss this kind of paper as "merely" engineering. I think this tendency is misguided. We must build giant machines and gather experimental evidence from them -- akin to physicists who build giant high-energy particle colliders to gather experimental evidence from them. The analogy is a bit off to me. As far as I can tell, there was significant impetus from within particle physics to commit a huge amount of resources and political effort toward verifying theories with experiment. I don't see anything similar in deep learning, because in this case the "theory" is mostly that "bigger is probably better". I think that idea is pretty uncontroversial for stuff like this. And if the work reduces to marshaling enough resources, what exactly is it? We should give OpenAI some credit for doing the damn thing, but as is the result kind of seems like an answer to a question that people weren't really asking.

by cs702 6y ago

100721 6y ago

I mostly agree with you. However, > It's a significant experimental accomplishment that points to a future in which general-purpose NLP models could be used for novel tasks without requiring additional training from the get-go. This premise is still purely science fiction. This model does not touch on either novel tasks nor being free from pretraining (unless I misunderstand). But overall, I think you’re right: it’s significant for a number of reasons.

cs702 6y ago
GPT-3 was pretrained on five datasets (Common Crawl, WebText2, Books1, Books2, and Wikipedia; see table 2.2), and then used on previously unseen tasks (Q&A, translation, cloze, etc.) without finetuning, i.e., weights were not updated after the original (autoregressive) pretraining. This promises a possible future in which general-purpose models are pretrained once, and deployed to production for multiple tasks.
- pacala 6y ago
  This is pretty mind-boggling, to the point of suspecting an error in the methodology. If it is just a language model, then it has no baked in notion of test time tasks. How on earth does a language model know what is required of it at test-time without fine tuning? How does it know that the test time prompt are examples of the task, and not some story prompt it's supposed to riff off in random ways?
  - cs702 6y ago
    Yes, mind-boggling. For each task, the authors feed a context window of text with either zero to a few sample queries and responses, followed by a query without the response. The model generates a response for the last query. BTW, this approach is analogous to what you would do with a human being: you would provide zero to a few sample questions and answers, and then ask a question.
    - deleted 6y ago
      [deleted]

chongli 6y ago

I'm reminded of Rich Sutton's essay, "The Bitter Lesson:" Moore's law is running on fumes at this point. The complexity of further scaling has reached geopolitical proportions. We need to get back to looking at more creative models in both the software and hardware domains. the model achieves competitive results on many NLP tasks and benchmarks without finetuning The article dismisses this result with the following analogy: “No, my 10-year-old math prodigy hasn’t proven any new theorems, but she can get a perfect score on the math SAT in under 10 minutes. Isn’t that groundbreaking?” And I tend to agree. We've had a game of benchmarking brinkmanship for a while now. At what point are we going to see some groundbreaking applications?

cs702 6y ago
I never said otherwise. Like physicists, we need both new theory (for new insights, ideas, models, etc.) and new experiments (for replication, performance, scalability, etc.) Also, Moore's Law is running on fumes, yes, but there's quite a bit of R&D focused on coming up with hardware that massively scales up (e.g., by more efficiently parallelizing) the dense and sparse multiply-sum operations common to so many AI models. I think Sutton's point about models that leverage computation is spot-on.
tal8d 6y ago
> We need to get back to looking at more creative models in both the software and hardware domains. That would be pretty foolish, given the fact that every hand crafted model eventually gets surpassed with brute force. A better use of time would be tackling whatever you mean by "complexity of further scaling has reached geopolitical proportions". I'm not a fan of it, as it is terribly inelegant, but denying the years of consistent brute force wins would just be silly. The best strategy for any nation with an interest in AI (be it economic or something much more skynety) would be securing two things very quickly: fabrication capacity and nuclear power, because this stuff is going to be measure megawatts - not ANN layers. Improving the efficiency of that conversion would certainly be helpful, but history has shown that to be a lower priority, just take a look at how ridiculously deep software stacks are compared to 20 years ago. I really wish the linguists had been proven right in the 1970s...

rmrfstar 6y ago

> practitioners with strong theoretical backgrounds tend to dismiss this kind of paper as "merely" engineering This paper implements an architecture that will be out of reach for me for about 5 years. So, I ask myself "why will this paper matter in 5 years?" There are two reasons I can imagine: 1. It shows that there is no phase change in the size-performance trend already documented over many orders of magnitude. 2. It was used as input data for pruning or distillation algorithms rooted in a better understanding of why language models work. If the NLP community remains laser focused on hill-climbing compute-agnostic benchmarks, I don't think there will be enough people working on 2. If 1 is the only reason it matters, I struggle to see how it is worth the cost in high-end research talent. It feels like standard low-risk corporate iteration.

cs702 6y ago
> "Why will this paper matter in 5 years?" 3. First evidence that performance continues to improve at hundreds of billions of weights -- paving the way for trillions of weights, approaching orders of magnitude comparable to that of the human brain connectome. 4. First evidence (AFAIK) that larger NLP models do not need task-specific finetuning -- paving the way for general-purpose models that work well on any NLP task without additional training. 5. First evidence (AFAIK) that larger NLP models fool human beings without cherry-picking -- paving the way for models that can pass ever more challenging Turing tests.
- deleted 6y ago
  [deleted]
jeeceebees 6y ago
I think the larger models get, the more incentive there is for researchers to look into pruning/distilling them for practical use. GPT-1,2,3 et al. have all shown that larger is better. While in the short term this means people will simply throw larger and larger clusters at the problem, in the longer term there needs to be inovation in making it more efficient on the clusters we have (as even the cloud has limits). I think sheer parameter count is an important part of the equation in general intelligence, so it's important that there are labs that work on scaling up promising leads to trillions of parameters on top of labs thinking of new promising directions.

Eridrus 6y ago

I'm an engineer and this is a boring paper. They didn't beat the conclusively beat SOTA the way BERT/T5 beat basically everything. Sure, this is "unsupervised", but from an engineering perspective that's entirely uninteresting, so we don't even get actual scaling numbers. This paper was written to pretend it is more than it is, which is basically the hallmark of everything OpenAI does.

The_rationalist 6y ago
There is one model that consistently outperform BERT: it's XLnet but it's already one year old, innovation seems to slow down

Al-Khwarizmi 6y ago

> More generally, I find that some AI researchers and practitioners with strong theoretical backgrounds tend to dismiss this kind of paper as "merely" engineering. It's just that this kind of work is more interesting as a general member of the public than as an AI researcher. As a human being I find it really interesting to see where this kind of models can take us. I was amazed playing with GPT-2 online demos and seeing to what extent it could generate text that looked like what a human could produce. With its quirks and problems, but still impressive. And I can't wait to put my fingers on a GPT-3 online demo. But as an NLP academic researcher (and this is not hypothetical, I'm actually one), what do I learn from this paper? What importance does it have to my research? Actually very little. You need more than 350 GB to fit the 175B parameters in memory, currently the largest GPU I can access has 24 GB (and I can access only one of those, which I use to -barely- run BERT-large). The cost of training the model in the cloud is estimated to be $12 million (https://twitter.com/eturner303/status/1266264358771757057 https://twitter.com/eturner303/status/1266264358771757057). This is a single training run, not including any neural architecture search, bug fixing, etc. So even though for an academic researcher my funding situation is not bad at all, I'm like a couple of orders of magnitude away from being able to do anything meaningful with models of this size, and can't expect that to change for at least 8-10 years (by which point, at the pace NLP evolves, this will be ancient history anyway). On the other hand, of course very often you learn useful ideas from papers that you can apply yourself even if it's not by implementing the same models in the paper, but that's not the case either. Here the lesson learned is "bigger is better" and I cannot train these enormous models, so there is not really much here that I can apply. So as an academic researcher, really there isn't a lot to do with this apart from shrugging, and basically dismissing it and just keeping trying to do our best with what we have. Which is still useful, at least if we don't want NLP applications to be in the hands of an oligopoly of megacorps and restricted only to the few most economically viable languages.

andi999 6y ago
I am doing other hpc stuff, so I am wondering why are you limited to 1 GPU? Windows recognizes up to 8 I think.
- 6y ago

KKKKkkkk1 6y ago

I don't know why this reminds you of Sutton's essay. Sutton claims that algorithmic work will be superseded by brute-force approaches powered by exponential growth in computing power. But this paper is not the result of exponential growth in computing power. It's the result of $1b worth of Azure credits provided for free by Microsoft. Rich Sutton is a great scientist but he is fooled by randomness. He initiated his research program just as Moore's law was taking off. Thanks to Moore, his approach saw incredible success and brought him deserved acclaim. But just as Moore's law is pulling the rug from under him, he is using his stature to claim that no other approach but his can work.

justicezyx 6y ago
The engineering effort of utilizing 1B worth of Azure credits is surely worth at least 10% of it.
- master_yoda_1 6y ago
  Yes then credit to azure engineers not open ai people
  - justicezyx 6y ago
    Wow, do you reasonably believe OpenAI people are not contributing?... I thought HN visitors are not capable of that level of naive thinking.
    - deleted 6y ago
      [deleted]
clickok 6y ago
You have misunderstood Sutton's argument. Quoting Rich: One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The point isn't that improvements in our algorithms is unnecessary or unhelpful, rather that the algorithms we should focus on will be capable of scaling with arbitrary amounts of compute/data. Such as, for example, neural networks, where we see an almost constant rate of improvement (for the appropriate architecture) as more resources are added.

gentleman11 6y ago

Think about the waves of fake news to come. Soon every corporation (and political group) in American will have a data center dedicated to overloading social media with messaging that benefits them. At least right now it’s not so hard to spot the bots. I am having some trouble seeing the benefits in lieu of the mess that is coming.

papeda 6y ago

> More generally, I find that some AI researchers and practitioners with strong theoretical backgrounds tend to dismiss this kind of paper as "merely" engineering. I think this tendency is misguided. We must build giant machines and gather experimental evidence from them -- akin to physicists who build giant high-energy particle colliders to gather experimental evidence from them. The analogy is a bit off to me. As far as I can tell, there was significant impetus from within particle physics to commit a huge amount of resources and political effort toward verifying theories with experiment. I don't see anything similar in deep learning, because in this case the "theory" is mostly that "bigger is probably better". I think that idea is pretty uncontroversial for stuff like this. And if the work reduces to marshaling enough resources, what exactly is it? We should give OpenAI some credit for doing the damn thing, but as is the result kind of seems like an answer to a question that people weren't really asking.