4 ms·
And so vanishing gradients are not a thing anymore?
by blackbear_ 2y ago
And so vanishing gradients are not a thing anymore?
- imjonse 2y agoGood question. That was an issue with tanh as activation function, and before residual connections and normalization layers. Tanh as a normalization but with other activations and residual present apparently is ok.
- tsurba 2y agoProper initialization is more important. Batch norm and others are important for faster convergence due to forcing the model to focus creating second and higher order nonlinearities, as a simple shift in mean/std is normalized out, and thus the gradient does not point in a direction that would only change those properties of the output distribution.
- tsurba 2y agoProper initialization of layers keeps gradient magnitudes from vanishing/exploding in deep networks. If you make sure the output of each layer has mean 0, std 1, the gradients will be reasonable as well, for example. I recommend e.g. the og resnet paper and its follow-up from Kaiming He et al. For a modern take on RNNs, read https://arxiv.org/abs/2303.06349 https://arxiv.org/abs/2303.06349 by DeepMind. There essentially the point is that largest eigenvalue (spectral radius) needs to be around 1, meaning repeated applications of a linear transformation doesn’t cause increase or decrease of the activations.
- blackbear_ 2y agoSure initialization helps, but are there also results about long term training dynamics? Even the paper you suggested had to use some sort of normalization to keep things stable
- toxik 2y agoTransformers learn residuals, as you can see in the figure. y = x + f(x).
- tripplyons 2y agoI think ResNet pretty much solved vanishing gradients. As for exploding gradients, that is typically with good parameter initialization and normalization. The paper in question proposes an alternative to normalization.