2 ms·
It makes its point well, but I'd like to see a followup paper addressing neural networks: given the extreme complexity of successful deep neural networks, which
by gwern 11y ago
It makes its point well, but I'd like to see a followup paper addressing neural networks: given the extreme complexity of successful deep neural networks, which outperform anything he considers there on real world problems he doesn't consider, what implications can we draw?
- Houshalter 11y agoNeural networks tend to overfit very easily, which is the main reason other methods usually outperform them. They are mainly successful where they can exploit the structure of a problem, in ways other methods can't. E.g. convolutional neural networks take advantage of local structure within images and the fact nearby pixels are related to each other. However I'd really like to see the reemergence of Bayesian neural networks, which can solve the overfitting problem. Also methods like dropout are relatively new, and alleviate overfitting a lot more than was possible in the past.