3 ms·

What if it was the following, with x = 0? y = x if x > 0 else -2 * x if x < 0 else 0

OisinMoran 9y ago
For functions that are not differentiable you can use the subdifferential [0] (or subgradient) which gives a set rather than a single value. In your example this would be the interval [-2,1]. In practice, from what I've read it seems to be fine to just take any value between these and most commonly they are just averaged. It helps that these regions are rarely encountered. For more detail, check out Chapter 6 [1], Section 6.3 "Hidden Units" in The Deep Learning book. [0] https://en.wikipedia.org/wiki/Subderivative https://en.wikipedia.org/wiki/Subderivative [1] http://www.deeplearningbook.org/contents/mlp.html http://www.deeplearningbook.org/contents/mlp.html
- mehrdadn 9y ago
  Nope. I thought about subderivative too... but then I realized this isn't even true. Constructing a counterexample is trivial: y = x if x > 0 else x if x < 0 else 0 Zero is in no way a subderivative at x = 0.
  - OisinMoran 9y ago
    Yeah you're right, the subdifferential of f(x)=|x| at the origin is the interval [−1, 1]. In the fourth paragraph of link [1] in the above post they explain quite well why we can get away with this type of trickery even if it seems wrong.
    - mehrdadn 9y ago
      > Yeah you're right, the subdifferential of f(x)=|x| at the origin is the interval [−1, 1]. I'm confused where you're seeing absolute value? I never had absolute value anywhere. The second example I wrote just amounted to y = x.
improbable22 9y ago
Clearly y'(0) = 0 here, since at x=0 the function is defined to be a constant. In calculus class this would be ill-defined / ambiguous, but the computer is simple-minded and picks one answer. This doesn't seem like a major flaw to me. What are you using this for which would care? Maybe you have something like a rectification which produces exactly x=0 much more often than 1 in 2^32... in which case a function like your y amounts to a special case for this. If you cared you could write a special case for the derivative too, here's one which will give y'(0)=42 but otherwise the automatic derivative: julia> y(x) = x>0 ? x : -2x julia> y(x::Dual) = x==0 ? dual(0, 42) : (x>0 ? x : -2x)
- ChrisRackauckas 9y ago
  Yeah, this is a good case to use multiple dispatch to flexibly tie into systems built on generic code. The problem with autodiff is it doesn't necessarily take all branches to know how to deal with these kinds of issues, but having the power to fix it yourself is crucial.
- JadeNB 9y ago
  > Clearly y'(0) = 0 here, since at x=0 the function is defined to be a constant. Every function is constant at every point. For example, I could define the identity function as `y ( x ) = if ( x > 0 ) { x } elsif ( x < 0 ) { x } else { 0 }`, but that doesn't mean that its derivative at the origin is `0`.
  - improbable22 9y ago
    No, dataflow's function y is defined in three smooth pieces, one of which is a constant. That's what this differentiation is seeing. It is evaluating at x+ϵ, and at 0+ϵ takes the ==0 branch which returns always 0, which has gradient 0. Your function also has three pieces, and the only difference from `y ( x ) = x` is that you've chosen to specify a derivative at zero, i.e. y(x+ϵ) = if ( x==0 ) { 0+0ϵ } else { x+1ϵ }. Which you may have reason to do, say to tell your optimiser to stop pushing on the wall.
deleted 9y ago
[deleted]

For functions that are not differentiable you can use the subdifferential [0] (or subgradient) which gives a set rather than a single value. In your example this would be the interval [-2,1]. In practice, from what I've read it seems to be fine to just take any value between these and most commonly they are just averaged. It helps that these regions are rarely encountered. For more detail, check out Chapter 6 [1], Section 6.3 "Hidden Units" in The Deep Learning book. [0] https://en.wikipedia.org/wiki/Subderivative https://en.wikipedia.org/wiki/Subderivative [1] http://www.deeplearningbook.org/contents/mlp.html http://www.deeplearningbook.org/contents/mlp.html

mehrdadn 9y ago
Nope. I thought about subderivative too... but then I realized this isn't even true. Constructing a counterexample is trivial: y = x if x > 0 else x if x < 0 else 0 Zero is in no way a subderivative at x = 0.
- OisinMoran 9y ago
  Yeah you're right, the subdifferential of f(x)=|x| at the origin is the interval [−1, 1]. In the fourth paragraph of link [1] in the above post they explain quite well why we can get away with this type of trickery even if it seems wrong.
  - mehrdadn 9y ago
    > Yeah you're right, the subdifferential of f(x)=|x| at the origin is the interval [−1, 1]. I'm confused where you're seeing absolute value? I never had absolute value anywhere. The second example I wrote just amounted to y = x.

improbable22 9y ago

Clearly y'(0) = 0 here, since at x=0 the function is defined to be a constant. In calculus class this would be ill-defined / ambiguous, but the computer is simple-minded and picks one answer. This doesn't seem like a major flaw to me. What are you using this for which would care? Maybe you have something like a rectification which produces exactly x=0 much more often than 1 in 2^32... in which case a function like your y amounts to a special case for this. If you cared you could write a special case for the derivative too, here's one which will give y'(0)=42 but otherwise the automatic derivative: julia> y(x) = x>0 ? x : -2x julia> y(x::Dual) = x==0 ? dual(0, 42) : (x>0 ? x : -2x)

ChrisRackauckas 9y ago
Yeah, this is a good case to use multiple dispatch to flexibly tie into systems built on generic code. The problem with autodiff is it doesn't necessarily take all branches to know how to deal with these kinds of issues, but having the power to fix it yourself is crucial.
JadeNB 9y ago
> Clearly y'(0) = 0 here, since at x=0 the function is defined to be a constant. Every function is constant at every point. For example, I could define the identity function as `y ( x ) = if ( x > 0 ) { x } elsif ( x < 0 ) { x } else { 0 }`, but that doesn't mean that its derivative at the origin is `0`.
- improbable22 9y ago
  No, dataflow's function y is defined in three smooth pieces, one of which is a constant. That's what this differentiation is seeing. It is evaluating at x+ϵ, and at 0+ϵ takes the ==0 branch which returns always 0, which has gradient 0. Your function also has three pieces, and the only difference from `y ( x ) = x` is that you've chosen to specify a derivative at zero, i.e. y(x+ϵ) = if ( x==0 ) { 0+0ϵ } else { x+1ϵ }. Which you may have reason to do, say to tell your optimiser to stop pushing on the wall.

deleted 9y ago

[deleted]