3 ms·
> Thousands of matrices multiplied together seems as if it would be hard to come to an actual understanding of... Most of the operations will cancel out. If I
by ars 10y ago
> Thousands of matrices multiplied together seems as if it would be hard to come to an actual understanding of...
Most of the operations will cancel out. If I multiply a number by a fixed series of numbers, even if that list is millions long, I can just pre-multiply them and have a single operation.
The same is likely happening here. Especially if all the operations are algerbraic it would not be hard to simplify and then analyze.
- rjbwork 10y agoYeah that makes sense I suppose. Just turning it all into a single matrix or an order of magnitude of smaller matrices would certainly simplify analysis.
- Houshalter 10y agoThe problem with neural networks is they use nonlinear transformations after every layer to prevent this. Usually something like max(0, x). However it is sometimes possible to train a much smaller neural network to imitate a larger network, or to prune irrelevant nodes and connections from a big network.
- rjbwork 10y agoDo they do it TO prevent it, or does it provide some additional benefit?
- Houshalter 10y agoWell sort of both. Because of the reasons mentioned above, a purely linear neural net wouldn't be able to learn anything more complicated than simple linear functions. Adding nonlinearities vastly increases the space of possible functions it can learn.
- rjbwork 10y agoAh right. Makes perfect sense. Cheers!