2 ms·
Ok, I see what you mean, and you're right. However, I'm not sure I see any usefulness of this property.
by p1esk 7y ago
Ok, I see what you mean, and you're right. However, I'm not sure I see any usefulness of this property.
- H8crilA 7y agoIt helps understand why NaNs or close to zero training speed happens. For example - say you have an LSTM. The more you unroll it the more matrices get multiplied on the way back, and the backprop signal can get amplified/muted in an exponential fashion (as a function of the number of unrolled cells).
- p1esk 7y agoI was talking about the property that gradients are linearly proportional to loss value. I don't see whether this property helps or hurts vanishing gradients problem (e.g. if it was not linear, maybe it would be worse).