4 ms·
This leads in a very interesting direction. Previously I read that DNN is just as content to fit random input as it is to fit meaningful categories of input. Th
by karmakaze 8y ago
This leads in a very interesting direction. Previously I read that DNN is just as content to fit random input as it is to fit meaningful categories of input. This makes it seem that the DNN is just a memory trick and can't possibly extract any meaning.
Now being able to compress the network after being trained, it's evident that not all the nodes were needed/used. A good experiment would be to train a DNN on random input samples and verify that it is far less compressible. If so, the degree of compressibily could be used as a proxy estimate of how much 'conceptualization' is being done.
- anishacharya 8y agoOvercomplete nature of DNN is a common phenomenon and often advocated to convert saddle points into local minima --- understanding the actual landscape is difficult due to the inherent nonlinearity.... this indeed leads to an interesting research direction and many prominent research groups have started investing heavily on this
- visarga 8y ago> Previously I read that DNN is just as content to fit random input as it is to fit meaningful categories of input. That's not exactly right. DNN's can memorise randomly given labels, but it's harder than learning real data, because real data has more regularities.