4 ms·
Doesn't have to be continuously differentiable to be differentiable right?
by mandarax8 3y ago
Doesn't have to be continuously differentiable to be differentiable right?
- Kevin09210 3y agoDon't downvote. Explain
- pvitz 3y agoHaven't downvoted, but the "continuously" in "continuously differentiable" refers to the derivative and not to the function itself. (Also, from differentiable follows continuous, but not the other way around (sufficient, but not necessary).)
- deleted 3y ago[deleted]
- littlestymaar 3y agoI need to be continuous to be differentiable. That being said, if it's only discontinuous in a few number of places you could extend the derivative everywhere by taking either the left or the right derivative, and then you'd end up with a gradient being defined everywhere, but not continuous. But then does gradient descent work if the gradient isn't continuous?
- amelius 3y agoYou can also extend the derivative such that you compute it numerically and then some slight discontinuities are certainly not a problem.
- PeterisP 3y agoOne of the common - if not the most common - activation function is 'rectified linear unit' (ReLU) which is a fancy name for y=max(x,0), which has a discontinuous gradient (1 if x>0, 0 if x<0) and that works mostly fine.