4 ms·
In the polynomial fit case: it’s pretty clear to me that if the number of parameters is higher than the degrees of freedom in the data, then the model can simpl
by clbrmbr 3y ago
In the polynomial fit case: it’s pretty clear to me that if the number of parameters is higher than the degrees of freedom in the data, then the model can simply memorize the input.
That doesn’t explain why it generalizes well, though.
- occamrazor 3y agoIt”s only mentioned at the end. If you use some regularization, you have many ways to fit the data (almost) perfectly, and you choose the one with the smallest coefficients, which is often one that generalizes well. And even without explicit regularization, some training methods implicitly apply sone form of regularization, when the number of parameters exceeds the number of data points.