4 ms·
The problem with neural networks is they use nonlinear transformations after every layer to prevent this. Usually something like max(0, x). However it is somet
by Houshalter 10y ago
The problem with neural networks is they use nonlinear transformations after every layer to prevent this. Usually something like max(0, x).
However it is sometimes possible to train a much smaller neural network to imitate a larger network, or to prune irrelevant nodes and connections from a big network.
- rjbwork 10y agoDo they do it TO prevent it, or does it provide some additional benefit?
- Houshalter 10y agoWell sort of both. Because of the reasons mentioned above, a purely linear neural net wouldn't be able to learn anything more complicated than simple linear functions. Adding nonlinearities vastly increases the space of possible functions it can learn.
- rjbwork 10y agoAh right. Makes perfect sense. Cheers!