3 ms·
I need to be continuous to be differentiable. That being said, if it's only discontinuous in a few number of places you could extend the derivative everywhere
by littlestymaar 3y ago
I need to be continuous to be differentiable.
That being said, if it's only discontinuous in a few number of places you could extend the derivative everywhere by taking either the left or the right derivative, and then you'd end up with a gradient being defined everywhere, but not continuous. But then does gradient descent work if the gradient isn't continuous?
- amelius 3y agoYou can also extend the derivative such that you compute it numerically and then some slight discontinuities are certainly not a problem.
- PeterisP 3y agoOne of the common - if not the most common - activation function is 'rectified linear unit' (ReLU) which is a fancy name for y=max(x,0), which has a discontinuous gradient (1 if x>0, 0 if x<0) and that works mostly fine.