3 ms·

> We’ll implement numerical differentiation, though in principle we could also implement symbolic differentiation I'd like to argue, that when it comes to comp

by sampo 3y ago

> We’ll implement numerical differentiation, though in principle we could also implement symbolic differentiation I'd like to argue, that when it comes to computers and differentiation, automatic differentiation is the most useful, most important in practice. But often only symbolic and numerical differentiation are mentioned.

version_five 3y ago
AD is important for training neural networks, or sgd (et al) generally. But that's still only one field. Numerical differentiation is still important e.g. for differential equation solvers. I don't think you can say AD is the most important or useful - maybe for understanding pop culture.
- thechao 3y ago
  AD has been a round since at least 1957 (the oldest reference to the class of technique I could find). When I studied it (from a CAS/PLT angle), it was considered a good middle-ground technique between symbolic methods & giving up. You can trace the result of the AD and recover a polynomial solution near the expected results (like a profile guided Taylor expansion, I guess?). It allowed us to run Buchberger's on algorithmic objects without analytic forms, and still have a chance at getting a complete basis for the antiderivative.
- sampo 3y ago
  > Numerical differentiation is still important e.g. for differential equation solvers. Is this a misunderstanding? You use differential equation solvers when all you know is how to calculate the derivative(s) of the function, and you want to get the function itself as the solution. Where would you need numerical differentiation in this?
- dls2016 3y ago
  Then we can combine AD and numerical solvers like is found in modern weather and climate models. I don't quite understand but it has something to do with sensitivity analysis and improving data assimilation. (Google "4dvar ecmwf" for more details... eg: https://www2.atmos.umd.edu/~dkleist/docs/da/ECMWF_DA_TUTORIAL/Benedetti-Tangent_linear_adjoint_models.pdf https://www2.atmos.umd.edu/~dkleist/docs/da/ECMWF_DA_TUTORIA...) I think the idea is to use the "tangent linear model" to decide how much importance to give to a particular observation of the initial state.
reikonomusha 3y ago
I think this is because automatic differentiation is a manifestation of just one rule of differentiation: the chain rule. With numerical differentiation, you only need to be able to evaluate the function at hand. With automatic differentiation, you need to "seed" your program with the differentials of all existing functions. What's nice about a discussion of symbolic differentiation is that we can prove a few rules rigorously, and then use those rules to purely mechanically differentiate algebraic expressions we encountered up to trigonometry. You're right though, in practice, for complex functions expressed as programs, automatic differentiation is superior.
- analog31 3y ago
  While in college, I wrote a program that did this. I thought I needed precise derivatives for a curve fitting program, so I wrote one that evaluated an expression and all of its derivatives, all at once. This was typical noobie chutzpah. First, I converted an infix expression into its postfix form using a textbook algorithm. Then I applied all of the differentiation rules to the postfix expression, including L'Hospital's rule. Was it correct? All I know is that I couldn't kill it.
suavesito 3y ago
That is because those are the traditional methods, and are easy to teach. And probably the curriculum hasn't been changed. Also, automatic differentiation is _hard_. Numerical differentiation comes directly from the definition of derivative and symbolic is what you always did for exercises in your calculus classes. Automatic differentiation come in two flavours, forward and backward modes. Forward mode is based on dual numbers [1], which is the quotient of a polynomial ring over the real numbers, by the principal ideal generated by the square on the indeterminate That is ℝ[X]/≺X²≻. Another way of thinking of this is to have an element ℇ different from zero, such that ℇ ²= 0 and hand wave the fact that the dual part of the number carry the derivative. Backward mode builds a graph of computation and doing symbolic differentiation over the graph and compile down the derivative into runable machine code (it could be interpreted, compiled down to IR, neither the form or the execution environment changes the fundamental algorithm). Maybe they are not really hard, but they are not easy either. Still I think they should be at least mentioned in modern courses. [1] https://en.wikipedia.org/wiki/Dual_number https://en.wikipedia.org/wiki/Dual_number
- JadeNB 3y ago
  > That is ℝ[X]/≺X²≻. > Another way of thinking of this is to have an element ℇ different from zero, such that ℇ ²= 0 and hand wave the fact that the dual part of the number carry the derivative. Aren't these the same way of thinking?
  - lupire 3y ago
    Different way of thinking about same thing. A more common example of this idea is √(-1): R[x]/<x²+1> vs i²+1=0.
- Jtsummers 3y ago
  > Still I think they [automatic differentiation methods] should be at least mentioned in modern courses. That might make more sense in another course, like one on numerical methods, rather than in a programming course that happens to use numerical differentiation as a demonstration of programming techniques and language features.
- 3y ago
burnished 3y ago
Holy shit, thanks for bringing that up, I had never heard of automatic differentiation. So cool!

by sampo 3y ago

version_five 3y ago

AD is important for training neural networks, or sgd (et al) generally. But that's still only one field. Numerical differentiation is still important e.g. for differential equation solvers. I don't think you can say AD is the most important or useful - maybe for understanding pop culture.

thechao 3y ago
AD has been a round since at least 1957 (the oldest reference to the class of technique I could find). When I studied it (from a CAS/PLT angle), it was considered a good middle-ground technique between symbolic methods & giving up. You can trace the result of the AD and recover a polynomial solution near the expected results (like a profile guided Taylor expansion, I guess?). It allowed us to run Buchberger's on algorithmic objects without analytic forms, and still have a chance at getting a complete basis for the antiderivative.
sampo 3y ago
> Numerical differentiation is still important e.g. for differential equation solvers. Is this a misunderstanding? You use differential equation solvers when all you know is how to calculate the derivative(s) of the function, and you want to get the function itself as the solution. Where would you need numerical differentiation in this?
dls2016 3y ago
Then we can combine AD and numerical solvers like is found in modern weather and climate models. I don't quite understand but it has something to do with sensitivity analysis and improving data assimilation. (Google "4dvar ecmwf" for more details... eg: https://www2.atmos.umd.edu/~dkleist/docs/da/ECMWF_DA_TUTORIAL/Benedetti-Tangent_linear_adjoint_models.pdf https://www2.atmos.umd.edu/~dkleist/docs/da/ECMWF_DA_TUTORIA...) I think the idea is to use the "tangent linear model" to decide how much importance to give to a particular observation of the initial state.

reikonomusha 3y ago

I think this is because automatic differentiation is a manifestation of just one rule of differentiation: the chain rule. With numerical differentiation, you only need to be able to evaluate the function at hand. With automatic differentiation, you need to "seed" your program with the differentials of all existing functions. What's nice about a discussion of symbolic differentiation is that we can prove a few rules rigorously, and then use those rules to purely mechanically differentiate algebraic expressions we encountered up to trigonometry. You're right though, in practice, for complex functions expressed as programs, automatic differentiation is superior.

analog31 3y ago
While in college, I wrote a program that did this. I thought I needed precise derivatives for a curve fitting program, so I wrote one that evaluated an expression and all of its derivatives, all at once. This was typical noobie chutzpah. First, I converted an infix expression into its postfix form using a textbook algorithm. Then I applied all of the differentiation rules to the postfix expression, including L'Hospital's rule. Was it correct? All I know is that I couldn't kill it.

suavesito 3y ago

That is because those are the traditional methods, and are easy to teach. And probably the curriculum hasn't been changed. Also, automatic differentiation is _hard_. Numerical differentiation comes directly from the definition of derivative and symbolic is what you always did for exercises in your calculus classes. Automatic differentiation come in two flavours, forward and backward modes. Forward mode is based on dual numbers [1], which is the quotient of a polynomial ring over the real numbers, by the principal ideal generated by the square on the indeterminate That is ℝ[X]/≺X²≻. Another way of thinking of this is to have an element ℇ different from zero, such that ℇ ²= 0 and hand wave the fact that the dual part of the number carry the derivative. Backward mode builds a graph of computation and doing symbolic differentiation over the graph and compile down the derivative into runable machine code (it could be interpreted, compiled down to IR, neither the form or the execution environment changes the fundamental algorithm). Maybe they are not really hard, but they are not easy either. Still I think they should be at least mentioned in modern courses. [1] https://en.wikipedia.org/wiki/Dual_number https://en.wikipedia.org/wiki/Dual_number

JadeNB 3y ago
> That is ℝ[X]/≺X²≻. > Another way of thinking of this is to have an element ℇ different from zero, such that ℇ ²= 0 and hand wave the fact that the dual part of the number carry the derivative. Aren't these the same way of thinking?
- lupire 3y ago
  Different way of thinking about same thing. A more common example of this idea is √(-1): R[x]/<x²+1> vs i²+1=0.
Jtsummers 3y ago
> Still I think they [automatic differentiation methods] should be at least mentioned in modern courses. That might make more sense in another course, like one on numerical methods, rather than in a programming course that happens to use numerical differentiation as a demonstration of programming techniques and language features.
3y ago

burnished 3y ago

Holy shit, thanks for bringing that up, I had never heard of automatic differentiation. So cool!