4 ms·

> it does so using a network with 480 million parameters. The training to ascertain the values of such a large number of parameters is even more remarkable bec

> it does so using a network with 480 million parameters. The training to ascertain the values of such a large number of parameters is even more remarkable because it was done with only 1.2 million labeled images—which may understandably confuse those of us who remember from high school algebra that we are supposed to have more equations than unknowns. This is one of the key misunderstandings are still deeply rooted in people's minds. For modern DL, a large part of the learning comes from "internal" data points, in this case the pixels of the image, as opposed to the labels. If you count the number of pixels, you will likely get something like 1.2 trillion, more than enough to justify the 4.8e8 parameters. It's the usage of internal data that prevents overfitting, NOT the random initialization and SGD as claimed in the article. Another way to see this is: if you need more labels than parameters, how can GPT3 have ANY parameters at all? It is trained purely on raw text data.

sendtown_expwy 5y ago
You are incorrect about the input dimensionality mattering. Let's say you have 100 high-res images with yes/no labels. If you hash the images and put their labels in a hashmap, you can say this is a "learned" function of 100 parameters which achieves zero training error on the dataset. This parameter count is independent of input dimension. Why do you think this would change when this mapping is replaced by a smooth neural network mapping? GPT is trained to predict the input (estimating p(x)), versus predicting a label given an input (p(y|x)). So in the case of GPT you can use the input dimensionality as a "label", as another responder has mentioned. ImageNet classification is different (excepting recent semi-supervised or unsupervised approaches to image recognition). The ability to generalize in the typical imagenet setting is, as the article says, a byproduct of SGD with early stopping, which in practice limits the number of functions a deep neural network can express (something not considered in an analysis which only considers parameter count).
- montenegrohugo 5y ago
  The point is your simple mapping with zero error on the training dataset also has zero prediction power in both the test dataset and in real life. It's learned nothing; it's at the extreme scale of overfitted. Input dimensionality is absolutely important when determining net size.
  - sendtown_expwy 5y ago
    That's the point. 100 parameters is sufficient to overfit, and it's a number that's independent of the input size. Do you have a reference for your statement?
    - montenegrohugo 5y ago
      Reference for what exactly? That input dimensionality is important when determining net size? That seems quite self-explanatory; try training a image classifier with only 100 parameters. Maybe I understood that question wrong, but regardless, even if early stopping wasn't implemented, a NN would have more predictive power than the hash mapping. Both would be completely overfit on the training data set, yet the NN would most likely be able to make some okay guesses with OOD data.
  - fspeech 5y ago
    Seems like cross talking to me. They were responding to the erroneous claim of "input dimensonlity" being equivalent to data. What the first poster referred to as "internal data points" may be better described as the presumption of differentiability, that is, a small disturbance of the pixels should result in a "small" change of the labels. But it was ridiculous to claim somehow the total number of pixels is a meaningful measure of sample size. The pixels are not independent, as dramatized by the hash map example given above.
contravariant 5y ago
That doesn't quite work out that way I think, if you compare it to solving a system of equations then the size of the input is irrelevant. Indeed a very large input is often the main reason for a problem to be under-specified. What you should look at is the number of outputs times the number of data points for each output. If this number is lower than the number of parameters then it should be possible to find multiple solutions. Of course in this case you're not looking for a solution, but an optimum, and not even a global one, so it's not too troubling per se that you don't get a unique answer. Though it does somewhat suggest you should be able to get an equivalent fit with far fewer parameters, but finding it could be quite tricky.
williamtrask 5y ago
GPT3 has millions of labels. Every vocabulary term is a label. It’s equivalent to supervised learning in architecture. The “self-supervised” business is mostly spin to make it sound a bit more novel. People have been predicting the next word for ages (Turing did this). Input: <previous words of article> Label: <next word> Your point is well taken that the number of input data points is also important when considering the complexity of the problem. In this case however the number of data points more or less exactly equals the number of labels. (About Me: the first year+ of my PhD was focused on large scale language modelling, during which transformers came out.)
axiosgunnar 5y ago
This is such a basic error the author has made, that I am not sure he can use „we“ when refering to researchers…
- commandlinefan 5y ago
  Well, that is something that you're taught in high school algebra, which you end up "unlearning" when you study linear algebra.
albertzeyer 5y ago
For modern DL, a large part also comes from regularization. And then also data augmentation. And self-supervision in whatever way, either prediction, masked prediction, contrastive losses, etc. Which all adds to the number of constraints / equations.
woopwoop 5y ago
Sanity check: you train a 48 million parameter image classification model on a single 4000x4000 rgb image. Do you overfit?
andreyk 5y ago
The whole point is weird - having more equations than unknowns is only needed for an exact closed form solution, it's entirely possible to solve underconstrained problems that have multiple solutions via optimization (which is what DL is). To be fair, the article says just that afterward, just weird that the high school thing was mentioned in the first place.

sendtown_expwy 5y ago

You are incorrect about the input dimensionality mattering. Let's say you have 100 high-res images with yes/no labels. If you hash the images and put their labels in a hashmap, you can say this is a "learned" function of 100 parameters which achieves zero training error on the dataset. This parameter count is independent of input dimension. Why do you think this would change when this mapping is replaced by a smooth neural network mapping? GPT is trained to predict the input (estimating p(x)), versus predicting a label given an input (p(y|x)). So in the case of GPT you can use the input dimensionality as a "label", as another responder has mentioned. ImageNet classification is different (excepting recent semi-supervised or unsupervised approaches to image recognition). The ability to generalize in the typical imagenet setting is, as the article says, a byproduct of SGD with early stopping, which in practice limits the number of functions a deep neural network can express (something not considered in an analysis which only considers parameter count).

montenegrohugo 5y ago
The point is your simple mapping with zero error on the training dataset also has zero prediction power in both the test dataset and in real life. It's learned nothing; it's at the extreme scale of overfitted. Input dimensionality is absolutely important when determining net size.
- sendtown_expwy 5y ago
  That's the point. 100 parameters is sufficient to overfit, and it's a number that's independent of the input size. Do you have a reference for your statement?
  - montenegrohugo 5y ago
    Reference for what exactly? That input dimensionality is important when determining net size? That seems quite self-explanatory; try training a image classifier with only 100 parameters. Maybe I understood that question wrong, but regardless, even if early stopping wasn't implemented, a NN would have more predictive power than the hash mapping. Both would be completely overfit on the training data set, yet the NN would most likely be able to make some okay guesses with OOD data.
- fspeech 5y ago
  Seems like cross talking to me. They were responding to the erroneous claim of "input dimensonlity" being equivalent to data. What the first poster referred to as "internal data points" may be better described as the presumption of differentiability, that is, a small disturbance of the pixels should result in a "small" change of the labels. But it was ridiculous to claim somehow the total number of pixels is a meaningful measure of sample size. The pixels are not independent, as dramatized by the hash map example given above.

contravariant 5y ago

That doesn't quite work out that way I think, if you compare it to solving a system of equations then the size of the input is irrelevant. Indeed a very large input is often the main reason for a problem to be under-specified. What you should look at is the number of outputs times the number of data points for each output. If this number is lower than the number of parameters then it should be possible to find multiple solutions. Of course in this case you're not looking for a solution, but an optimum, and not even a global one, so it's not too troubling per se that you don't get a unique answer. Though it does somewhat suggest you should be able to get an equivalent fit with far fewer parameters, but finding it could be quite tricky.

williamtrask 5y ago

GPT3 has millions of labels. Every vocabulary term is a label. It’s equivalent to supervised learning in architecture. The “self-supervised” business is mostly spin to make it sound a bit more novel. People have been predicting the next word for ages (Turing did this). Input: <previous words of article> Label: <next word> Your point is well taken that the number of input data points is also important when considering the complexity of the problem. In this case however the number of data points more or less exactly equals the number of labels. (About Me: the first year+ of my PhD was focused on large scale language modelling, during which transformers came out.)

axiosgunnar 5y ago

This is such a basic error the author has made, that I am not sure he can use „we“ when refering to researchers…

commandlinefan 5y ago
Well, that is something that you're taught in high school algebra, which you end up "unlearning" when you study linear algebra.

albertzeyer 5y ago

For modern DL, a large part also comes from regularization. And then also data augmentation. And self-supervision in whatever way, either prediction, masked prediction, contrastive losses, etc. Which all adds to the number of constraints / equations.

woopwoop 5y ago

Sanity check: you train a 48 million parameter image classification model on a single 4000x4000 rgb image. Do you overfit?

andreyk 5y ago

The whole point is weird - having more equations than unknowns is only needed for an exact closed form solution, it's entirely possible to solve underconstrained problems that have multiple solutions via optimization (which is what DL is). To be fair, the article says just that afterward, just weird that the high school thing was mentioned in the first place.