> What would a satisfactory "why" even look like exactly? Before deep learning happened, neural networks used to be popular for regression and interpolation pr

> What would a satisfactory "why" even look like exactly? Before deep learning happened, neural networks used to be popular for regression and interpolation problems. For a long time, our understanding of how they worked wasn't much better than our current understanding of how deep learning works. In 1994, Radford Neal showed [1] that weight-decay neural networks were in fact an approximation to Bayesian inference on a Gaussian process prior over the space of possible functions. Amongst other benefits, this allowed better approximations to be utilised, and essentially (along with the advent of SVMs) marked the end the first neural network era. Something like that is what I would consider a satisfactory "why" [1] http://www.cs.toronto.edu/~radford/pin.abstract.html http://www.cs.toronto.edu/~radford/pin.abstract.html