4 ms·
[citation needed] Sure, there's a huge amount of research happening in the field right now. But you make it sound like there's a ton of low hanging fruit, whic
by andbberger 11y ago
[citation needed]
Sure, there's a huge amount of research happening in the field right now. But you make it sound like there's a ton of low hanging fruit, which is emphatically not true.
Also, freezing the representations in the bottom layers usually doesn't lead to very good results. The representation the bottom layers learn in the standard formulation of deep feedforward networks is informed by gradient information in the output layers[1]. If you stop training the bottom layers after some time you're sacrificing representational power and in fact increasing the ultimate amount of computation that will need to performed to train the network to some level.
In fact, in a recent paper [2] some folks at Google describe how they achieved great performance training a huge CNN by inserting temporary output layers in the middle of the network while training (among other things). This increased the amount of gradient information that was propagated back to the bottom layers, forcing them to learn more powerful representations.
[1] https://en.wikipedia.org/wiki/Backpropagation https://en.wikipedia.org/wiki/Backpropagation
[2] http://arxiv.org/abs/1409.4842 http://arxiv.org/abs/1409.4842
- mytochar 11y agoRegarding backpropagation and training sections of the NN at different times, there are other training algorithms. Evolutionary training algorithms come to mind, and you could really evolve any section you wanted. You could even train the output of each layer one by one to represent a certain form of input to the future layer.
- darkmighty 11y agoThose training algorithms should accomplish the same goal, but backpropagation (or some variant) turns out to be the best choice for the kind of network shown here (large, non-recurrent), because it directly uses the learning gradient, while evolutionary methods don't usually make use of this information. I'm not a researcher in this area, but I'm quite sure you'd have a bad time going with pure evolutionary methods for this kind of network.
- andbberger 11y agoVirtually everyone uses gradient based methods in the end to fine tune the weights. Yes, there are other methods. Contrastive divergence seems to be king right now - of note is Minimum probability flow learning [1] (of which CD is a special case of). However the flavor of these methods tends to be tuning the weights of the model in such a way to maximize how close the model comes to sharing the probability distribution of the data. One can generally not constraint the model parameters (ie by freezing a layer) and retain the models ability to 'learn' the data distribution. [1]http://arxiv.org/abs/0906.4779 http://arxiv.org/abs/0906.4779
- raverbashing 11y agoWhile it's true there's a lot of research, using ReLu instead of tanh and sigmoid was a pretty much low-hanging fruit that took a lot of time to be applied