4 ms·
The answer is always "No it doesn't need all those nodes", but we do it that way because it makes doing the math easier. I bet they just removed a bunch of edge
by blamestross 3y ago
The answer is always "No it doesn't need all those nodes", but we do it that way because it makes doing the math easier. I bet they just removed a bunch of edge weights to nodes, but used the same matrices for calculating output.
I wish arbitrary topology networks were scalable (I love NEAT) but bipartide graphs crunch good in GPUs.
- zaptrem 3y agoCan you elaborate what you mean by "edge weights to nodes, but used the same matrices"?
- feanaro 3y agoThe network retains the same topology, and therefore the same matrix dimensions, but some elements are set to zero, removing their contribution.
- AlexErrant 3y agoRelevant excerpt from https://news.ycombinator.com/item?id=38656172 https://news.ycombinator.com/item?id=38656172 > ...if we meet the shape 512x512 (512 is a power of 2, therefore a very "round" number in computer science), then maybe some kernel will be very fast; but then on a 512x511matrix, the same kernel may need to add some padding first to transform it into a round 512x512 matrix with zeros at the end of each row. Adding those zeros means shifting all rows, which is a very costly operation.
- eurekin 3y agoI'm quite surprised so much was retained; many presentations on convolutional networks suggest adding a lot of features, but prune most in a post learning step. Supposedly lots of features, which are initialized randomly, just raise the probability of being close to the real answer (the concept being learned). Once it's found, most of features typically go unused or are reduntant.