3 ms·
For functions that are not differentiable you can use the subdifferential [0] (or subgradient) which gives a set rather than a single value. In your example thi
by OisinMoran 8y ago
For functions that are not differentiable you can use the subdifferential [0] (or subgradient) which gives a set rather than a single value. In your example this would be the interval [-2,1].
In practice, from what I've read it seems to be fine to just take any value between these and most commonly they are just averaged. It helps that these regions are rarely encountered.
For more detail, check out Chapter 6 [1], Section 6.3 "Hidden Units" in The Deep Learning book.
[0] https://en.wikipedia.org/wiki/Subderivative https://en.wikipedia.org/wiki/Subderivative
[1] http://www.deeplearningbook.org/contents/mlp.html http://www.deeplearningbook.org/contents/mlp.html
- mehrdadn 8y agoNope. I thought about subderivative too... but then I realized this isn't even true. Constructing a counterexample is trivial: y = x if x > 0 else x if x < 0 else 0 Zero is in no way a subderivative at x = 0.
- OisinMoran 8y agoYeah you're right, the subdifferential of f(x)=|x| at the origin is the interval [−1, 1]. In the fourth paragraph of link [1] in the above post they explain quite well why we can get away with this type of trickery even if it seems wrong.
- mehrdadn 8y ago> Yeah you're right, the subdifferential of f(x)=|x| at the origin is the interval [−1, 1]. I'm confused where you're seeing absolute value? I never had absolute value anywhere. The second example I wrote just amounted to y = x.