5 ms·

Did you try taking the derivative of y = cuberoot(x^3) at 0? If it works fine, tell me your secret!

FooBarBizBazz 7y ago
Thank you, this is interesting. I just tried. You need L'Hospital's Rule. I'd realized this was an issue with forward-mode autodiff, but hadn't thought about reverse-mode to realize it comes up there too. In general, NaN is a placeholder for "all the derivatives you didn't know you needed in advance". I think the Dual Numbers shed light on this (see "Division": [https://en.wikipedia.org/wiki/Dual_number#Division] https://en.wikipedia.org/wiki/Dual_number#Division]). One could imagine lazily computing higher derivatives as needed for L'Hospital's Rule in either Forward or Reverse mode, however. For Forward Mode, I guess you'd replace your eager "Dual Number" pairs with lazy lists? For reverse mode it seems you could do extra "passes" on the tape as needed... People who do this stuff for a living must know(?), but a quick Google doesn't turn up an answer for me...
- FooBarBizBazz 7y ago
  Here's the (pedantically-worked) example with reverse-mode: Program: y = g(x) = x^3 z = f(y) = cuberoot(y) Want: dz/dx Reverse-mode autodiff: // Forward pass y = g(0) = 0^3 = 0 g'(0) = 3 x^2 = 0 z = f(0) = cuberoot(0) = 0 f'(0) = (1/3) / cuberoot(y^2) = (1/3)/0 = Inf // Reverse pass dz/dy = f'(0) = Inf dz/dx = dz/dy * g'(0) = Inf * 0 = NaN
- FooBarBizBazz 7y ago
  Actually I'm wrong about L'Hospital's Rule in this problem; it solves nothing here. I would insert an "[EDIT ...]" into my original post to avoid misleading people, but it's too late to edit. It doesn't matter how many times you differentiate x^(1/3); you can not evaluate the result at zero. Which is funny, because you'd think you could differentiate x^3 and use the inverse function theorem. But there's that caveat about it not working when the derivative is zero. I am now revisiting high school calculus and doubting my own sanity. Will post updates if/when I get my head right.
adamnemecek 7y ago
Sorry, I was out. Ok, f(x) = cuberoot(x^3) f(0 + eps) = cuberoot((0 + eps)^3) = cuberoot(eps^3) = eps e.g. value is 0 and the slope is 1.
- gugagore 7y ago
  What is eps^3? ;) You have to turn that into some representation on the computer before you can even pass it to cuberoot.
  - adamnemecek 7y ago
    Theres is no representation of that value.In this case power is defined only in the range 1-2. You need to simplify the roots first.
- petschge 7y ago
  except by the definition of dual numbers eps^2 is 0, as is eps^3. So the slope you computed is 0, which is rather wrong for (a complicated representation of) y=x.
  - adamnemecek 7y ago
    Eps^3 is undefined. I don't see a problem with simplifying the exponents first. > a complicated representation of) y=x. The problem started as that. So no surprise.
    - deleted 7y ago
      [deleted]
    - tntn 7y ago
      > I don't see a problem with simplifying the exponents first. Sure, we call that computer algebra. As soon as you start doing that, you aren't doing automatic differentiation, you are doing (at least in part) symbolic differentiation.

Thank you, this is interesting. I just tried. You need L'Hospital's Rule. I'd realized this was an issue with forward-mode autodiff, but hadn't thought about reverse-mode to realize it comes up there too. In general, NaN is a placeholder for "all the derivatives you didn't know you needed in advance". I think the Dual Numbers shed light on this (see "Division": [https://en.wikipedia.org/wiki/Dual_number#Division] https://en.wikipedia.org/wiki/Dual_number#Division]). One could imagine lazily computing higher derivatives as needed for L'Hospital's Rule in either Forward or Reverse mode, however. For Forward Mode, I guess you'd replace your eager "Dual Number" pairs with lazy lists? For reverse mode it seems you could do extra "passes" on the tape as needed... People who do this stuff for a living must know(?), but a quick Google doesn't turn up an answer for me...

FooBarBizBazz 7y ago
Here's the (pedantically-worked) example with reverse-mode: Program: y = g(x) = x^3 z = f(y) = cuberoot(y) Want: dz/dx Reverse-mode autodiff: // Forward pass y = g(0) = 0^3 = 0 g'(0) = 3 x^2 = 0 z = f(0) = cuberoot(0) = 0 f'(0) = (1/3) / cuberoot(y^2) = (1/3)/0 = Inf // Reverse pass dz/dy = f'(0) = Inf dz/dx = dz/dy * g'(0) = Inf * 0 = NaN
FooBarBizBazz 7y ago
Actually I'm wrong about L'Hospital's Rule in this problem; it solves nothing here. I would insert an "[EDIT ...]" into my original post to avoid misleading people, but it's too late to edit. It doesn't matter how many times you differentiate x^(1/3); you can not evaluate the result at zero. Which is funny, because you'd think you could differentiate x^3 and use the inverse function theorem. But there's that caveat about it not working when the derivative is zero. I am now revisiting high school calculus and doubting my own sanity. Will post updates if/when I get my head right.

adamnemecek 7y ago

Sorry, I was out. Ok, f(x) = cuberoot(x^3) f(0 + eps) = cuberoot((0 + eps)^3) = cuberoot(eps^3) = eps e.g. value is 0 and the slope is 1.

gugagore 7y ago
What is eps^3? ;) You have to turn that into some representation on the computer before you can even pass it to cuberoot.
- adamnemecek 7y ago
  Theres is no representation of that value.In this case power is defined only in the range 1-2. You need to simplify the roots first.
petschge 7y ago
except by the definition of dual numbers eps^2 is 0, as is eps^3. So the slope you computed is 0, which is rather wrong for (a complicated representation of) y=x.
- adamnemecek 7y ago
  Eps^3 is undefined. I don't see a problem with simplifying the exponents first. > a complicated representation of) y=x. The problem started as that. So no surprise.
  - deleted 7y ago
    [deleted]
  - tntn 7y ago
    > I don't see a problem with simplifying the exponents first. Sure, we call that computer algebra. As soon as you start doing that, you aren't doing automatic differentiation, you are doing (at least in part) symbolic differentiation.