4 ms·
> Now ReLU is continuous itself (as well as being monotonic) so it still cooperates relatively well with gradient descent algorithms. A function with a discont
by KKKKkkkk1 4y ago
> Now ReLU is continuous itself (as well as being monotonic) so it still cooperates relatively well with gradient descent algorithms.
A function with a discontinuous derivative cannot cooperate with gradient descent algorithms. That's why you have the famous problem of "dead neurons".
- taylorius 4y agoDead neurons do not come about because of ReLU's discontinuous gradient. It is because for large regions of parameter space (i<0) the gradient is 0.
- KKKKkkkk1 4y ago> Dead neurons do not come about because of ReLU's discontinuous gradient. It is because for large regions of parameter space (i<0) the gradient is 0. If they had a continuous derivative that would not be the case.
- taylorius 4y agoImagine an alternative ReLU which had a narrow curved section, to smooth out the discontinuity. Now it has a continuous derivative, but that gradient is still zero for values < 0. This flat region is the cause of dead neurons, because backprop multiplies the propagated error by the gradient, to update the weight - and if the gradient is 0, the result of the multiplication is 0 , and the neuron's weights do not get adjusted. A derivative can be continuous, and zero.