4 ms·

Pardon me for asking what might be an obvious answer to some, but does increasing the parameters lead to a linear growth of the capability of the neural net, or

Pardon me for asking what might be an obvious answer to some, but does increasing the parameters lead to a linear growth of the capability of the neural net, or is it different. My knowledge of neural nets and AI is just lacking.

Dwedit 3y ago
There is a chart that compares Parameter Count, Size of model in GB, and "Perplexity". (Size of model is on a logarithmic scale) https://user-images.githubusercontent.com/48489457/243093269-07aa49f0-4951-407f-9789-0b5a01ce95b8.png https://user-images.githubusercontent.com/48489457/243093269... You can see that "Perplexity" goes down as Model Size goes up.
ianbutler 3y ago
There was some recent research that was confirming capabilities improve smoothly as a function of parameter count yes. Their point was a lot of research that had been showing jumps in performance at certain "breakpoints" for lack of a better word were the result of badly selected metrics versus a case of suddenly emergent behaviour. The nice thing about that research is it suggests that if you are able to try something on a smaller model it will scale nicely to a bigger model.
- sigmoid10 3y ago
  This has been known for several years. Especially zero and few-shot task performance scales extremely well with number of parameters. But more recently it was shown that you can actually trade parameters with training data volume and training time as well once you go into the billions of parameters. So while it takes more time to train, you can habe an equally powerful model with much fewer parameters and thus faster inference times.
  - deleted 3y ago
    [deleted]
  - ianbutler 3y ago
    The paper was fairly recent, https://arxiv.org/abs/2304.15004 https://arxiv.org/abs/2304.15004. It was more thoroughly confirming what was generally agreed upon while debunking other reasons. The key insight from the abstract, "Specifically, nonlinear or discontinuous metrics produce apparent emergent abilities, whereas linear or continuous metrics produce smooth, continuous predictable changes in model performance." Yup that is another recent and interesting development for sure!
    - sigmoid10 3y ago
      The authors are a bit disingenuous here. They insinuate that GPT3's performance shows unpredictable behaviour change at certain scales using their weirdly constructed metrics (which may or may not be true - see below), while the original GPT3 paper already showed how these amazing "emergent" capabilities scale with parameters in a very predictable way: https://arxiv.org/pdf/2005.14165.pdf https://arxiv.org/pdf/2005.14165.pdf Also note that the plots in the appendix contain some obvious errors, so you definitely want to wait for a peer reviewed version of this paper (if it ever survives review).
      - ianbutler 3y ago
        Sorry, I think you've misunderstood, they're saying that's exactly the point. Those weird metrics are what they're debunking, not supporting. Per my snippet from the abstract in my last comment. Their point was a lot of papers use those weird metrics and it contributes to the appearance of emergent ability, when in reality its just the bad metrics. Nothing you've said so far disagrees with either my understanding or the conclusion of the paper I linked.
        sigmoid10 3y ago
        I think you misunderstood. The authors created the very issue they are "debunking." They took GPT3, slapped on some random metrics and showed that these metrics don't show scaling behaviour correctly, while the original publication of GPT3 actually did it correctly in the first place.
- diffeomorphism 3y ago
  The figures in the paper you linked below are log_10 scale on one axis but not the other. Thus even without quite understanding what all these figures measure, it seems to me that answer the question about "linear growth" is a hard no: double the size will not be twice as good. Or am I misunderstanding how to read these graphs?
haldujai 3y ago
Complicated question, the answer is “it depends”. Several factors influence performance beyond parameter count, notable ones include: training corpus quality, training flops, and the downstream task. It depends on how much compute you are spending on training and how big of a model you’re talking about. There’s a “minimum” tokens/parameter for increasing size to be effective at improving loss/perplexity, so as you go up in parameters you generally will have to broaden your corpus which may lower it’s quality (e.g. tweets/reddit posts vs books/articles). This effect isn’t as significant at 65B parameters as there is still enough high quality training data but if you’re talking 1T the corpus will (probably, I haven’t tried this/seen this done) by necessity be significantly poorer quality as you will overfit by simply repeating (it’s only beneficial so many times to repeat). As a general rule, when validation loss/perplexity are the same in two models of different sizes downstream performance seems to be also generally the same (this was briefly explored in the PaLM 2 paper by Google) although it doesn’t correlate perfectly. Practically speaking, this translates into a bigger model is better for applications we’re generally talking about. It just may not hold infinitely which we’re starting to see evidence of. It’s definitely not linear though, you can look at some of the OpenLLaMA benchmarks (without getting into the weeds of if current benchmarks are representative) and the accuracy improvements in even 13B is not that significant (noting here all models were trained for the smear number of tokens, so relatively overtraining the smaller models). There are probably some threshold parameter sizes that make a big difference but it’s still being determined. https://ai.google/static/documents/palm2techreport.pdf https://ai.google/static/documents/palm2techreport.pdf
brucethemoose2 3y ago
What others said ^ But note thst Meta's LLaMA 33B and 65B were trained with more tokens (1.2 trillion?) than the 13B and 7B models (1 trillion). And subjectively, the larger parameter models do indeed feel "smarter" beyond what objective metrics would suggest.

Dwedit 3y ago

There is a chart that compares Parameter Count, Size of model in GB, and "Perplexity". (Size of model is on a logarithmic scale) https://user-images.githubusercontent.com/48489457/243093269-07aa49f0-4951-407f-9789-0b5a01ce95b8.png https://user-images.githubusercontent.com/48489457/243093269... You can see that "Perplexity" goes down as Model Size goes up.

ianbutler 3y ago

There was some recent research that was confirming capabilities improve smoothly as a function of parameter count yes. Their point was a lot of research that had been showing jumps in performance at certain "breakpoints" for lack of a better word were the result of badly selected metrics versus a case of suddenly emergent behaviour. The nice thing about that research is it suggests that if you are able to try something on a smaller model it will scale nicely to a bigger model.

sigmoid10 3y ago
This has been known for several years. Especially zero and few-shot task performance scales extremely well with number of parameters. But more recently it was shown that you can actually trade parameters with training data volume and training time as well once you go into the billions of parameters. So while it takes more time to train, you can habe an equally powerful model with much fewer parameters and thus faster inference times.
- deleted 3y ago
  [deleted]
- ianbutler 3y ago
  The paper was fairly recent, https://arxiv.org/abs/2304.15004 https://arxiv.org/abs/2304.15004. It was more thoroughly confirming what was generally agreed upon while debunking other reasons. The key insight from the abstract, "Specifically, nonlinear or discontinuous metrics produce apparent emergent abilities, whereas linear or continuous metrics produce smooth, continuous predictable changes in model performance." Yup that is another recent and interesting development for sure!
  - sigmoid10 3y ago
    The authors are a bit disingenuous here. They insinuate that GPT3's performance shows unpredictable behaviour change at certain scales using their weirdly constructed metrics (which may or may not be true - see below), while the original GPT3 paper already showed how these amazing "emergent" capabilities scale with parameters in a very predictable way: https://arxiv.org/pdf/2005.14165.pdf https://arxiv.org/pdf/2005.14165.pdf Also note that the plots in the appendix contain some obvious errors, so you definitely want to wait for a peer reviewed version of this paper (if it ever survives review).
    - ianbutler 3y ago
      Sorry, I think you've misunderstood, they're saying that's exactly the point. Those weird metrics are what they're debunking, not supporting. Per my snippet from the abstract in my last comment. Their point was a lot of papers use those weird metrics and it contributes to the appearance of emergent ability, when in reality its just the bad metrics. Nothing you've said so far disagrees with either my understanding or the conclusion of the paper I linked.
      - sigmoid10 3y ago
        I think you misunderstood. The authors created the very issue they are "debunking." They took GPT3, slapped on some random metrics and showed that these metrics don't show scaling behaviour correctly, while the original publication of GPT3 actually did it correctly in the first place.
diffeomorphism 3y ago
The figures in the paper you linked below are log_10 scale on one axis but not the other. Thus even without quite understanding what all these figures measure, it seems to me that answer the question about "linear growth" is a hard no: double the size will not be twice as good. Or am I misunderstanding how to read these graphs?

haldujai 3y ago

Complicated question, the answer is “it depends”. Several factors influence performance beyond parameter count, notable ones include: training corpus quality, training flops, and the downstream task. It depends on how much compute you are spending on training and how big of a model you’re talking about. There’s a “minimum” tokens/parameter for increasing size to be effective at improving loss/perplexity, so as you go up in parameters you generally will have to broaden your corpus which may lower it’s quality (e.g. tweets/reddit posts vs books/articles). This effect isn’t as significant at 65B parameters as there is still enough high quality training data but if you’re talking 1T the corpus will (probably, I haven’t tried this/seen this done) by necessity be significantly poorer quality as you will overfit by simply repeating (it’s only beneficial so many times to repeat). As a general rule, when validation loss/perplexity are the same in two models of different sizes downstream performance seems to be also generally the same (this was briefly explored in the PaLM 2 paper by Google) although it doesn’t correlate perfectly. Practically speaking, this translates into a bigger model is better for applications we’re generally talking about. It just may not hold infinitely which we’re starting to see evidence of. It’s definitely not linear though, you can look at some of the OpenLLaMA benchmarks (without getting into the weeds of if current benchmarks are representative) and the accuracy improvements in even 13B is not that significant (noting here all models were trained for the smear number of tokens, so relatively overtraining the smaller models). There are probably some threshold parameter sizes that make a big difference but it’s still being determined. https://ai.google/static/documents/palm2techreport.pdf https://ai.google/static/documents/palm2techreport.pdf

brucethemoose2 3y ago

What others said ^ But note thst Meta's LLaMA 33B and 65B were trained with more tokens (1.2 trillion?) than the 13B and 7B models (1 trillion). And subjectively, the larger parameter models do indeed feel "smarter" beyond what objective metrics would suggest.