2 ms·
It helps understand why NaNs or close to zero training speed happens. For example - say you have an LSTM. The more you unroll it the more matrices get multiplie
by H8crilA 7y ago
It helps understand why NaNs or close to zero training speed happens. For example - say you have an LSTM. The more you unroll it the more matrices get multiplied on the way back, and the backprop signal can get amplified/muted in an exponential fashion (as a function of the number of unrolled cells).
- p1esk 7y agoI was talking about the property that gradients are linearly proportional to loss value. I don't see whether this property helps or hurts vanishing gradients problem (e.g. if it was not linear, maybe it would be worse).