3 ms·
Hmm interesting, in the well-known book Neural Networks and Deep Learning [1] by Michael Nielsen, it is shown that neural networks can approximate any function
by mettamage 8y ago
Hmm interesting, in the well-known book Neural Networks and Deep Learning [1] by Michael Nielsen, it is shown that neural networks can approximate any function [2].
This leaves the question: are neural networks able to produce more functions than simple polynomials? Apparently the answer of the summary of the mentioned paper is no (I didn't read the actual paper, only the summary).
I simply only know the basics of neural networks (I fully understand chapter 1 and 2 of [1]), so I wouldn't know understand why the two are practically equivalent. I mean, I can imagine why but I'd be making the fallacy of simply understanding that a neural net is able to approximate any polynomial and then think "yup, that's it, no clue how to generate counter examples, QED for all I care O:)"
So with this in mind is anyone able to argue why neural nets are essentially only polynomial regression?
[1] http://neuralnetworksanddeeplearning.com/ http://neuralnetworksanddeeplearning.com/
[2] http://neuralnetworksanddeeplearning.com/chap4.html http://neuralnetworksanddeeplearning.com/chap4.html
- jefft255 8y agoThey are not "equivalent". Polynomials can only produce polynomials, but these polynomials can be made to be as close as you want to any continuous function (see Stone-Weierstrass theorem). For example, a NN with a ReLU activation function produce a function that is not a polynomial. Now I understand that this not the main point; both polynomials and NN can approximate any function that is not artificially nasty. The only thing is that a NN is easier to optimize and generalizes better for high dimensional problems such as image classification. The issue is more complicated than simply expressiveness. The author claim that "one should use PR instead" seems dubious at best, but I'm now going to read the paper and figure it out.