3 ms·
To be honest, I have never understood all the hoopla concerning the universal approximation theorem. Least squares in your favorite orthormal basis for L2[0,1]
by hyperbovine 12y ago
To be honest, I have never understood all the hoopla concerning the universal approximation theorem. Least squares in your favorite orthormal basis for L2[0,1] will do the same thing and it can be easier to match the correct basis with your problem domain. If there was some result showing that NNs converge at a faster rate (as a function of the number of parameters being fitted) that would be nice, but I am not aware of any. Also, universal approximation has nothing to do with Turing completeness.
- abhgh 12y agoWith the right basis functions, yes, you would end up approximating the target function well. The problem is how do you determine the right basis functions? In SVMs, for example, there is no one good way to pick the right kernel today. This is a benefit with ANNs - the universal approximation theorem tells you that there is nothing else you need to decide on once you're using ANNs. Sure there are parameters like the no of layers etc, but those parameters are something you need account for in other techniques too. Not saying ANNs give the best solution always. Or even the best first model to try out on all problems; only pointing out that there is a convincing case to be made for them, and the universal approximation theorem has a lot to do with it. Wrt your other point, Turing completeness was proved separately - look for the paper "Turing computability with neural nets" by Siegelmann and Sontag. Not sure your parent comment intended the association.