4 ms·

That makes more sense. The title is flat out wrong IMO.

adgjlsfhk1 1y ago
I think it is correct. lots of people view AD as a black box that you can throw algorithms to and get derivatives out, and this shows that that isn't true
- wakawaka28 1y ago
  If you wrote code that failed to compile, you wouldn't impulsively call your compiler incorrect. This title sounds like it puts the blame in the wrong place. You can get error accumulation from even a basic calculation in a loop. We could try to solve these problems but it's not the algorithm's fault you don't know what you're doing.
  - ChrisRackauckas 1y ago
    This has nothing to do with floating point error accumulation or numerical stability in the floating point sense. You can do this with arbitrary sized floating point values and you will still get the same non-convergence result.
    - wakawaka28 1y ago
      Yes, it does. I admit that I have not tried to get deep into this, but it says right in the summary that this is due to error propagation and numerical instability. You can have such errors in ordinary hand-written code. The size of the float does not matter (much) for some examples, which wipe out a bunch of significant figures from the result. I'm not going to sit here and pretend I know exactly the details going on here but I studied numerical analysis and AD quite a bit back in the day. Even in the 2010s people knew that there was a chance of the resulting expressions having inherent instability. Even simple things like the quadratic formula have more stable and less stable forms. How much worse might it be for auto-generated expressions (or expression sequences, or equivalent; AD can be done a few ways) such as in AD? AD guarantees analytically correct logic (in infinite precision, for example) if you use it right but error analysis is not even attempted by most libraries.
      - adgjlsfhk1 1y ago
        > AD guarantees analytically correct logic (in infinite precision, for example) if you use it right The entire point of the video is that this isn't true. It is true for static algorithms, but for algorithms that iterate to convergence, the AD will ensure that the primal has converged, but will not ensure the dual has converged.
        wakawaka28 1y ago
        I don't think you understood what I wrote there. The rules for algebraically computing a derivative are simple and deterministic and essentially captured within AD algorithms. They MUST therefore be correct in an analytical sense, given infinite precision. The video starts off by saying, they are dealing with considerations about how computers actually work. That kind of implies finite precision. Like I said, concerns about stability of the methods are not new. Your original function might not be differentiable at any given point, for example. You have to know about that stuff rather than blindly applying "automatic" techniques. There is a lot of literature about how to use AD and what can go wrong. This is a paper I just found in a basic search, that is a survey of known pitfalls: https://wires.onlinelibrary.wiley.com/doi/full/10.1002/widm.1555 https://wires.onlinelibrary.wiley.com/doi/full/10.1002/widm.... The entire point of the video is mired in an hour or so of details about how they had trouble using it for solving ODEs. I am familiar with forward and reverse mode but for me to appreciate it I would have to get up to speed with their exact problem and terminology. Anyway, my point is that AD requires you to know what you are doing. This video seems like a valuable contribution to the state of the art but I think you have to recognize that the potential for problems was known to numerical analysis experts for decades, so this is not as groundbreaking as it appears. The title should read, "Automatic differentiation can be tricky to use" to establish that it is in fact a skill issue. The mitigation of these corner cases is valuable, to make them more versatile or foolproof. But the algorithms are not incorrect just because you didn't get it to solve your problem.
        wakawaka28 1y ago
        Not to spam you, but this is probably a function that would not work for AD: https://math.stackexchange.com/questions/2383397/differentiability-of-the-sum-of-sum-n-1-infty-frac-sinnxn2 https://math.stackexchange.com/questions/2383397/differentia... That is, it is a series that converges, but trying to take the derivative as a sum of individual terms results in divergence. I learned a lot of this type stuff ages ago, but in 2025 I just searched for an example lol... I am long overdue for a review of numerical analysis and real analysis. ChatGPT also says something about an example related to some Fourier series, maybe related to this: https://en.m.wikipedia.org/wiki/Convergence_of_Fourier_series https://en.m.wikipedia.org/wiki/Convergence_of_Fourier_serie... You can ask it all about this stuff. It seems pretty decent, although I have not gone too far into it.

I think it is correct. lots of people view AD as a black box that you can throw algorithms to and get derivatives out, and this shows that that isn't true

wakawaka28 1y ago
If you wrote code that failed to compile, you wouldn't impulsively call your compiler incorrect. This title sounds like it puts the blame in the wrong place. You can get error accumulation from even a basic calculation in a loop. We could try to solve these problems but it's not the algorithm's fault you don't know what you're doing.
- ChrisRackauckas 1y ago
  This has nothing to do with floating point error accumulation or numerical stability in the floating point sense. You can do this with arbitrary sized floating point values and you will still get the same non-convergence result.
  - wakawaka28 1y ago
    Yes, it does. I admit that I have not tried to get deep into this, but it says right in the summary that this is due to error propagation and numerical instability. You can have such errors in ordinary hand-written code. The size of the float does not matter (much) for some examples, which wipe out a bunch of significant figures from the result. I'm not going to sit here and pretend I know exactly the details going on here but I studied numerical analysis and AD quite a bit back in the day. Even in the 2010s people knew that there was a chance of the resulting expressions having inherent instability. Even simple things like the quadratic formula have more stable and less stable forms. How much worse might it be for auto-generated expressions (or expression sequences, or equivalent; AD can be done a few ways) such as in AD? AD guarantees analytically correct logic (in infinite precision, for example) if you use it right but error analysis is not even attempted by most libraries.
    - adgjlsfhk1 1y ago
      > AD guarantees analytically correct logic (in infinite precision, for example) if you use it right The entire point of the video is that this isn't true. It is true for static algorithms, but for algorithms that iterate to convergence, the AD will ensure that the primal has converged, but will not ensure the dual has converged.
      - wakawaka28 1y ago
        I don't think you understood what I wrote there. The rules for algebraically computing a derivative are simple and deterministic and essentially captured within AD algorithms. They MUST therefore be correct in an analytical sense, given infinite precision. The video starts off by saying, they are dealing with considerations about how computers actually work. That kind of implies finite precision. Like I said, concerns about stability of the methods are not new. Your original function might not be differentiable at any given point, for example. You have to know about that stuff rather than blindly applying "automatic" techniques. There is a lot of literature about how to use AD and what can go wrong. This is a paper I just found in a basic search, that is a survey of known pitfalls: https://wires.onlinelibrary.wiley.com/doi/full/10.1002/widm.1555 https://wires.onlinelibrary.wiley.com/doi/full/10.1002/widm.... The entire point of the video is mired in an hour or so of details about how they had trouble using it for solving ODEs. I am familiar with forward and reverse mode but for me to appreciate it I would have to get up to speed with their exact problem and terminology. Anyway, my point is that AD requires you to know what you are doing. This video seems like a valuable contribution to the state of the art but I think you have to recognize that the potential for problems was known to numerical analysis experts for decades, so this is not as groundbreaking as it appears. The title should read, "Automatic differentiation can be tricky to use" to establish that it is in fact a skill issue. The mitigation of these corner cases is valuable, to make them more versatile or foolproof. But the algorithms are not incorrect just because you didn't get it to solve your problem.
      - wakawaka28 1y ago
        Not to spam you, but this is probably a function that would not work for AD: https://math.stackexchange.com/questions/2383397/differentiability-of-the-sum-of-sum-n-1-infty-frac-sinnxn2 https://math.stackexchange.com/questions/2383397/differentia... That is, it is a series that converges, but trying to take the derivative as a sum of individual terms results in divergence. I learned a lot of this type stuff ages ago, but in 2025 I just searched for an example lol... I am long overdue for a review of numerical analysis and real analysis. ChatGPT also says something about an example related to some Fourier series, maybe related to this: https://en.m.wikipedia.org/wiki/Convergence_of_Fourier_series https://en.m.wikipedia.org/wiki/Convergence_of_Fourier_serie... You can ask it all about this stuff. It seems pretty decent, although I have not gone too far into it.