5 ms·

> By all accounts, deep neural networks like VGG have way too many parameters and should overfit. But they don’t. This isn't true at all. You can only make thi

> By all accounts, deep neural networks like VGG have way too many parameters and should overfit. But they don’t. This isn't true at all. You can only make this statement relative to a dataset. And VGG is almost always trained on ImageNet. A standard VGG16 has 138 million parameters. ImageNet has 1 million images each is 244x244 (~50k). There are ~431 input data points per parameter in VGG16. VGG is not overparameterized for ImageNet. There are plenty of network/dataset pairs they could have given as an example, but this isn't it.

marcinzm 5y ago
VGG also has regularization (L2, dropout, maxpool) if I remember correctly so it's sort of odd to use that as an example of overfitting when it explicitly tunes against that. I'm guessing the epochs and learning rate were also tuned to lower overfitting. The same is true to every other popular network out there.
derbOac 5y ago
I have so many questions about this now. For example, how often are people not distinguishing between amount of data versus number of cases? Do we really have any sense of how "effective number of parameters" scales with network features? This article had me wondering if peoples' understanding of these things is backward or something.
- light_hue_1 5y ago
  > I have so many questions about this now. For example, how often are people not distinguishing between amount of data versus number of cases? To state your question more broadly. Data has structure that's why we can learn in the first place. Can we separately talk about the structure vs number of examples of the different structures in our data? In computer vision this would be sort of like talking about how much information is within an object class vs between object class. We don't know. And we don't have the mathematical tools to talk about this today. > Do we really have any sense of how "effective number of parameters" scales with network features? This article had me wondering if peoples' understanding of these things is backward or something. It's not backward, it's non-existent. We basically try things and when we have too many parameters we see that performance isn't increasing (or isn't increasing enough to be worth it, or with unsupervised methods and massive datasets we just max out the size of network we can train).
PeterisP 5y ago
Wait, shouldn't you measure the information in output data (information about classes of samples) to evaluate overparameterization, not all the information in various features? I.e. total overfitting can be achieved by ignoring most of the input values and using just as many pixels from the top left corner as you need to uniquely identify the source image (which is far less than the total number of pixels) and map it to an arbitrary class, and for a million images that should be theoretically possible with million-ish parameters; i.e. no matter how many pixels there were in each image, 138 million parameters for classification of a million images is overparameterized from the classic perspective of what overparameterization is (some generalization forced simply because it's theoretically impossible to fit each sample individually).

marcinzm 5y ago

VGG also has regularization (L2, dropout, maxpool) if I remember correctly so it's sort of odd to use that as an example of overfitting when it explicitly tunes against that. I'm guessing the epochs and learning rate were also tuned to lower overfitting. The same is true to every other popular network out there.

derbOac 5y ago

I have so many questions about this now. For example, how often are people not distinguishing between amount of data versus number of cases? Do we really have any sense of how "effective number of parameters" scales with network features? This article had me wondering if peoples' understanding of these things is backward or something.

light_hue_1 5y ago
> I have so many questions about this now. For example, how often are people not distinguishing between amount of data versus number of cases? To state your question more broadly. Data has structure that's why we can learn in the first place. Can we separately talk about the structure vs number of examples of the different structures in our data? In computer vision this would be sort of like talking about how much information is within an object class vs between object class. We don't know. And we don't have the mathematical tools to talk about this today. > Do we really have any sense of how "effective number of parameters" scales with network features? This article had me wondering if peoples' understanding of these things is backward or something. It's not backward, it's non-existent. We basically try things and when we have too many parameters we see that performance isn't increasing (or isn't increasing enough to be worth it, or with unsupervised methods and massive datasets we just max out the size of network we can train).

PeterisP 5y ago

Wait, shouldn't you measure the information in output data (information about classes of samples) to evaluate overparameterization, not all the information in various features? I.e. total overfitting can be achieved by ignoring most of the input values and using just as many pixels from the top left corner as you need to uniquely identify the source image (which is far less than the total number of pixels) and map it to an arbitrary class, and for a million images that should be theoretically possible with million-ish parameters; i.e. no matter how many pixels there were in each image, 138 million parameters for classification of a million images is overparameterized from the classic perspective of what overparameterization is (some generalization forced simply because it's theoretically impossible to fit each sample individually).