3 ms·
The advantage of neural networks is that they can be trained. You can give it a set of inputs and desired outputs, and do gradient descent on it. Neural networ
by Houshalter 11y ago
The advantage of neural networks is that they can be trained. You can give it a set of inputs and desired outputs, and do gradient descent on it.
Neural networks are essentially like trainable digital circuits. The proof of universality shows that neurons can approximate any kind of logic gate (or any input output mapping, like a lookup table.) A lookup table by itself isn't terribly useful, but you can put them in a series and make arbitrary circuits. And that makes (deep) neural networks strictly more powerful than "shallow" methods.
- Retric 11y agoA very important caveat is the ability to be trained + universality does not mean they can be trained to fit any function to arbitrary precision in finite time.
- Houshalter 11y agoWell of course not. If you could fit any neural network to any function quickly, you would have super powers. But in practice, local optima do seem to stop being an issue in big neural networks. Also this article shows a method of how to construct a lookup table from a neural network in linear time. So in the worst case you can just memorize the input to output table, quickly. In the best case you can fit a simple elegant model, which fits the data perfectly with very few parameters. Given unlimited amounts of time to search the parameter space. Real world NNs are somewhere between these two.