4 ms·
> which is a weakness in the theory because how can more neurons cause more problems? In exactly the same way that adding more terms to a polynomial fit causes
by dfan 10y ago
> which is a weakness in the theory because how can more neurons cause more problems?
In exactly the same way that adding more terms to a polynomial fit causes more problems. The is one of the most fundamental results in the theory of statistical learning in general; don't blame Deep Learning for it.
- amelius 10y agoYes I know, it was a rhetorical question. Imho, if having more parameters causes problems, then the system should simply not use those extra parameters. But the theory is not there yet.
- dfan 10y agoThat's what regularization is for. You probably know that too, so pretend that was just for the benefit of the onlookers.
- cityhall 10y agoI think his point is no one can tell you from theory which regularization methods to apply to a particular problem to get the best results. You need expert knowledge, experience, and hyperparameter tuning.
- prats226 10y agoTransfer learning helps with overfitting too. It is proven to get more generalized model if you use transfer learning that if you train a model on your own with same data (even with large datasets). You need expertise in deep learning but the good thing is that you don't need a lot of expertise in domain of the problem.
- elyase 10y agoIt can be argued that some algorithms like Random Forests don't impose a generalization penalty as you increase the number of parameters (forests).
- apathy 10y agoRF is appallingly difficult to re-use for inference, though. At least with a DNN or CNN you can pop open the hood and see what the model is doing at various points. Tradeoffs, tradeoffs everywhere. It's almost like traditional mathematical statistics has something to offer them fancy machine learners. (Breiman was a professor of statistics, after all... ahead of his time, but no less a statistician.)