4 ms·
> Deep-learning models are overparameterized, which is to say they have more parameters than there are data points available for training. Is this true for all
by djoldman 5y ago
> Deep-learning models are overparameterized, which is to say they have more parameters than there are data points available for training.
Is this true for all deep learning models?
- lvl100 5y agoDL cannot be over-specified. However you do need to mind your endogenous and exogenous variables.
- jjcon 5y agoNot even close, most of my work has been in naturally occurring data and there is way waay more data available than can possibly be used (petabytes). Where they get this idea as being the rule and not the exception is beyond me.
- dekhn 5y agodepends on the model, but most systemns I've worked with had millions to billions of parameters, and trillions of (sparsely populated) data points.
- armoredkitten 5y agoIt's not inherently true. Technically, deep learning is essentially any neural network model with hidden layers (i.e., one layer in between the input layer and the output layer). You could have a "deep learning" model with a couple dozen parameters, perhaps. But at that end of the scale, most people would probably reach for other approaches that are more easily interpretable (e.g., logistic regression, random forest). So in practice, yes, virtually any deep learning model you see out there in the wild, even most "toy examples" used to teach machine learning, are going to be overparameterized.