2 ms·
https://smerity.com/articles/2016/google_nmt_arch.html https://smerity.com/articles/2016/google_nmt_arch.html section "V6: Residuals are the new hotness". You c
by sp332 10y ago
https://smerity.com/articles/2016/google_nmt_arch.html https://smerity.com/articles/2016/google_nmt_arch.html section "V6: Residuals are the new hotness". You can think of each layer (or I guess a pair of layers) as trying to correct errors in the previous layer. If the previous layer is close enough, the next layer can output 0. This is more stable than a straight linear activation, where one layer might move up a bit as the next one moves down to compensate! ReLU lets the earlier layer fine-tune itself without the effect being undone by later layers.
- cr0sh 10y agoInteresting! Thanks for the link!