5 ms·

Autodiff is insane. Kids waste how much time in school doing analytical calculus like troglodytes.

by adamnemecek 7y ago

taneq 7y ago
You... do realise why doing these things analytically is worth while, right?
- adamnemecek 7y ago
  I dont, no.
  - wolfram74 7y ago
    Numerical methods can be efficient, but nothing beats an analytic method that can be simply evaluated after derived. Often times you can go from an approximate answer with numerous function evaluations to an exact answer in one simpler function evaluation. Pathological cases not withstanding.
    - adamnemecek 7y ago
      AD is NOT a numerical method, it's very precise. You can in fact do it on paper as well. https://blog.demofox.org/2014/12/30/dual-numbers-automatic-differentiation/ https://blog.demofox.org/2014/12/30/dual-numbers-automatic-d... Analytical derivatives grow exponentially longer with the length of the expression. AD doesn't have this problem.
      - tntn 7y ago
        Just because automatic differentiation is a different technique than numerical differentiation doesn't mean it isn't a numerical method. (I probably should not say this, but I will anyway: your comments regarding math and science always give me the distinct impression that you know the name of a lot of things without knowing much about the thing. You are frequently breathless about analog quantum computing, Lie algebras, Chu spaces, some universal concept of duality, but then you fail to respond with substantial arguments to people that try to temper the breathlessness. If you want to convince people of things, you should work on this.)
        adamnemecek 7y ago
        > Just because automatic differentiation is a different technique than numerical differentiation doesn't mean it isn't a numerical method. It does mean that. Pls see the overview paper http://jmlr.org/papers/volume18/17-468/17-468.pdf http://jmlr.org/papers/volume18/17-468/17-468.pdf in particular section "2.1 AD Is Not Numerical Differentiation" > substantial arguments to people that try to temper the breathlessness Which arguments in particular? > our comments regarding math and science always give me the distinct impression that you know the name of a lot of things without knowing much about the thing. Ok be more specific, which ones do you care about? Maybe I don't always have time to answer everyone. > some universal concept of duality It's the concept of adjointness. And yes, it's very universal.
        tntn 7y ago
        Huh, interesting. First time I read this comment, it said something like "why should I care what you think?" I guess the edit indicates that you do care?
        adamnemecek 7y ago
        No, I edited it because HN has reached out to me many times before that I need to be less of an asshole or I'll get banned. So no, I don't care about what you think, but I don't want to get banned. Anyway, nice that you read the first version but still didn't respond to the other questions.
  - formalsystem 7y ago
    I don't understand why this is being downvoted. It's not like doing division in long form is that meaningful.
    - petschge 7y ago
      It is still important to know, even if most division problems you encounter are handed of to a calculator or computer. Why? Because understanding long division is a required background for understanding polynomial division. And understanding that is required to understand how you can get all root of a polynomial using Newtons method. And that is required to figure out how many different plasma modes there are. And THAT is a problem no computer can simply answer for you. Of course only a small number of people care and we don't teach about plasma modes in school, but we teach the basics that you need to start your way towards more complicated things. Long division of polynomials is also used in the construction of CRC sums. So no, understanding long division is not meaningless. It give you a starting point for a lot of other things.
noelwelsh 7y ago
I don't really understand this comment. The rules of forward mode automatic differentiation are exactly the rules of symbolic differentiation one would learn in high school, except AD is evaluated at a particular point (thus avoiding the need to build an expression tree and thus avoiding exponential blow-up in the size of said tree). AD is also on fairly shaky ground once you step outside what can be expressed in standard analytical calculus. For example it is fairly easy to express the same function in different ways and get derivatives that differ. A lot of work on semantics is needed.
- adamnemecek 7y ago
  > For example it is fairly easy to express the same function in different ways and get derivatives that differ. Example?
  - gugagore 7y ago
    I posted an example. y=cuberoot(x^3) and y=x should give different derivatives at x=0 in, I'm guessing, all autodiff approaches you'll get your hands on.
    - adamnemecek 7y ago
      How does y=x break anything? It works fine.
      - gugagore 7y ago
        Did you try taking the derivative of y = cuberoot(x^3) at 0? If it works fine, tell me your secret!
        FooBarBizBazz 7y ago
        Thank you, this is interesting. I just tried. You need L'Hospital's Rule. I'd realized this was an issue with forward-mode autodiff, but hadn't thought about reverse-mode to realize it comes up there too. In general, NaN is a placeholder for "all the derivatives you didn't know you needed in advance". I think the Dual Numbers shed light on this (see "Division": [https://en.wikipedia.org/wiki/Dual_number#Division] https://en.wikipedia.org/wiki/Dual_number#Division]). One could imagine lazily computing higher derivatives as needed for L'Hospital's Rule in either Forward or Reverse mode, however. For Forward Mode, I guess you'd replace your eager "Dual Number" pairs with lazy lists? For reverse mode it seems you could do extra "passes" on the tape as needed... People who do this stuff for a living must know(?), but a quick Google doesn't turn up an answer for me...
brighter2morrow 7y ago
Learning to do calculus allows you to do it on paper and in your head. Lines of code are liabilities that need to be maintained, understood concepts are assets that let you build, maintain, and simplify.
- adamnemecek 7y ago
  AD can be done on paper. Analytical calculus derivatives get exponentially longer with the length of the derived expression. https://blog.demofox.org/2014/12/30/dual-numbers-automatic-differentiation/ https://blog.demofox.org/2014/12/30/dual-numbers-automatic-d...
gugagore 7y ago
It's good to question what we teach. Like I never learned any methods for computing a square root, to arbitrary precision, whereas my parents did. But autodiff doesn't replace symbolic differentiation. Consider the derivative of cuberoot(x^3). Autodiff struggles at x=0 because x^3 is flat (slope 0) and cuberoot is vertical (slope inf). And together, with autodiff, you get slope NaN. If you wanted to try to fix that problem with autodiff, well first off it's likely not worth it because similar problems are inherent to autodiff, but you'll need an analytic understanding of what's going on.
- adamnemecek 7y ago
  I'm out right now but this doesn't sound right. How does autodiff struggle with it?
  - gugagore 7y ago
    You will have to put a bit of work into thinking about it, if you don't already see it. At the risk of being more rude than effective, I spelled it out pretty clearly above if you understand autodiff, which requires an understanding of the chain rule, which most people first learn about in a calculus class. If you want to claim that autodiff is a replacement for all our differentiation needs, then you should try to understand when it doesn't work.
  - tobmlt 7y ago
    The differentiation of a fractional power (fraction less than 1.) involves a negative fractional power. 0 raised to some negative power involves dividing by zero. This gives you some kind of NAN in, e.g., vanilla languages such as Python. The "issue with automatic differentiation" is really an issue with dividing by zero. Graph x(1./3.). It's slope is vertical at zero.
    - gugagore 7y ago
      I don't think this is right. An autodiff package will likely report +inf for the derivative of cuberoot at 0. The problem is really that there are a lot of ways the function can have infinite slope. The sqrt function also has infinite slope at 0. Dividing by zero isn't where the NaN comes from.
eli_gottlieb 7y ago
Autodiff only works because analytical calculus can prove that differentiation is a functor over functions.
- jjtheblunt 7y ago
  over continuous functions?
  - eli_gottlieb 7y ago
    Well technically, over differentiable functions, but yes. Derivatives can be evaluated by decomposing functions, taking elementary derivatives, and recomposing the functions, in a way that integrals can't be.
nthompson 7y ago
Automatic differentiation only works on functions which are locally complex analytic. It fails on things that fall outside that model, but are still differentiable. Daubechies wavelets are a good example of this.
- ChrisRackauckas 7y ago
  Julia's systems can make use of ChainRules.jl which doesn't assume locally complex analytic, and instead uses its Wirtinger derivatives (df/dz and df/dzbar), and if complex analytic just uses the well-known simplification df/dzbar=0. This allows for non-analytic functions to be used, and you just need to supply all primitives in Wirtinger form. Julia has enough non-ML people using this stuff that complex numbers actually get used here :).
deleted 7y ago
[deleted]
petschge 7y ago
Kids in school say "why do I to learn X, I am never going to need it in real live". And then you end up with adults who don't believe that the earth is round, that vaccines are low-risk and extremely beneficial and think that the existence of snow means that there is no climate change and that homeopathy and essential oils are a valid alternative to modern medicine, etc.
- adamnemecek 7y ago
  This is completely unrelated but nice job that you put me in the same group as flat earthers. Because, yes, if you complain about current calculus curriculum, you are not just a flat earther but also LITERALLY HITLER. It's not about learning x, it's about the approaches you learn to solve x.

Autodiff is insane. Kids waste how much time in school doing analytical calculus like troglodytes.

taneq 7y ago

You... do realise why doing these things analytically is worth while, right?

adamnemecek 7y ago
I dont, no.
- wolfram74 7y ago
  Numerical methods can be efficient, but nothing beats an analytic method that can be simply evaluated after derived. Often times you can go from an approximate answer with numerous function evaluations to an exact answer in one simpler function evaluation. Pathological cases not withstanding.
  - adamnemecek 7y ago
    AD is NOT a numerical method, it's very precise. You can in fact do it on paper as well. https://blog.demofox.org/2014/12/30/dual-numbers-automatic-differentiation/ https://blog.demofox.org/2014/12/30/dual-numbers-automatic-d... Analytical derivatives grow exponentially longer with the length of the expression. AD doesn't have this problem.
    - tntn 7y ago
      Just because automatic differentiation is a different technique than numerical differentiation doesn't mean it isn't a numerical method. (I probably should not say this, but I will anyway: your comments regarding math and science always give me the distinct impression that you know the name of a lot of things without knowing much about the thing. You are frequently breathless about analog quantum computing, Lie algebras, Chu spaces, some universal concept of duality, but then you fail to respond with substantial arguments to people that try to temper the breathlessness. If you want to convince people of things, you should work on this.)
      - adamnemecek 7y ago
        > Just because automatic differentiation is a different technique than numerical differentiation doesn't mean it isn't a numerical method. It does mean that. Pls see the overview paper http://jmlr.org/papers/volume18/17-468/17-468.pdf http://jmlr.org/papers/volume18/17-468/17-468.pdf in particular section "2.1 AD Is Not Numerical Differentiation" > substantial arguments to people that try to temper the breathlessness Which arguments in particular? > our comments regarding math and science always give me the distinct impression that you know the name of a lot of things without knowing much about the thing. Ok be more specific, which ones do you care about? Maybe I don't always have time to answer everyone. > some universal concept of duality It's the concept of adjointness. And yes, it's very universal.
        tntn 7y ago
        Huh, interesting. First time I read this comment, it said something like "why should I care what you think?" I guess the edit indicates that you do care?
        adamnemecek 7y ago
        No, I edited it because HN has reached out to me many times before that I need to be less of an asshole or I'll get banned. So no, I don't care about what you think, but I don't want to get banned. Anyway, nice that you read the first version but still didn't respond to the other questions.
- formalsystem 7y ago
  I don't understand why this is being downvoted. It's not like doing division in long form is that meaningful.
  - petschge 7y ago
    It is still important to know, even if most division problems you encounter are handed of to a calculator or computer. Why? Because understanding long division is a required background for understanding polynomial division. And understanding that is required to understand how you can get all root of a polynomial using Newtons method. And that is required to figure out how many different plasma modes there are. And THAT is a problem no computer can simply answer for you. Of course only a small number of people care and we don't teach about plasma modes in school, but we teach the basics that you need to start your way towards more complicated things. Long division of polynomials is also used in the construction of CRC sums. So no, understanding long division is not meaningless. It give you a starting point for a lot of other things.

noelwelsh 7y ago

I don't really understand this comment. The rules of forward mode automatic differentiation are exactly the rules of symbolic differentiation one would learn in high school, except AD is evaluated at a particular point (thus avoiding the need to build an expression tree and thus avoiding exponential blow-up in the size of said tree). AD is also on fairly shaky ground once you step outside what can be expressed in standard analytical calculus. For example it is fairly easy to express the same function in different ways and get derivatives that differ. A lot of work on semantics is needed.

adamnemecek 7y ago
> For example it is fairly easy to express the same function in different ways and get derivatives that differ. Example?
- gugagore 7y ago
  I posted an example. y=cuberoot(x^3) and y=x should give different derivatives at x=0 in, I'm guessing, all autodiff approaches you'll get your hands on.
  - adamnemecek 7y ago
    How does y=x break anything? It works fine.
    - gugagore 7y ago
      Did you try taking the derivative of y = cuberoot(x^3) at 0? If it works fine, tell me your secret!
      - FooBarBizBazz 7y ago
        Thank you, this is interesting. I just tried. You need L'Hospital's Rule. I'd realized this was an issue with forward-mode autodiff, but hadn't thought about reverse-mode to realize it comes up there too. In general, NaN is a placeholder for "all the derivatives you didn't know you needed in advance". I think the Dual Numbers shed light on this (see "Division": [https://en.wikipedia.org/wiki/Dual_number#Division] https://en.wikipedia.org/wiki/Dual_number#Division]). One could imagine lazily computing higher derivatives as needed for L'Hospital's Rule in either Forward or Reverse mode, however. For Forward Mode, I guess you'd replace your eager "Dual Number" pairs with lazy lists? For reverse mode it seems you could do extra "passes" on the tape as needed... People who do this stuff for a living must know(?), but a quick Google doesn't turn up an answer for me...

brighter2morrow 7y ago

Learning to do calculus allows you to do it on paper and in your head. Lines of code are liabilities that need to be maintained, understood concepts are assets that let you build, maintain, and simplify.

adamnemecek 7y ago
AD can be done on paper. Analytical calculus derivatives get exponentially longer with the length of the derived expression. https://blog.demofox.org/2014/12/30/dual-numbers-automatic-differentiation/ https://blog.demofox.org/2014/12/30/dual-numbers-automatic-d...

gugagore 7y ago

It's good to question what we teach. Like I never learned any methods for computing a square root, to arbitrary precision, whereas my parents did. But autodiff doesn't replace symbolic differentiation. Consider the derivative of cuberoot(x^3). Autodiff struggles at x=0 because x^3 is flat (slope 0) and cuberoot is vertical (slope inf). And together, with autodiff, you get slope NaN. If you wanted to try to fix that problem with autodiff, well first off it's likely not worth it because similar problems are inherent to autodiff, but you'll need an analytic understanding of what's going on.

adamnemecek 7y ago
I'm out right now but this doesn't sound right. How does autodiff struggle with it?
- gugagore 7y ago
  You will have to put a bit of work into thinking about it, if you don't already see it. At the risk of being more rude than effective, I spelled it out pretty clearly above if you understand autodiff, which requires an understanding of the chain rule, which most people first learn about in a calculus class. If you want to claim that autodiff is a replacement for all our differentiation needs, then you should try to understand when it doesn't work.
- tobmlt 7y ago
  The differentiation of a fractional power (fraction less than 1.) involves a negative fractional power. 0 raised to some negative power involves dividing by zero. This gives you some kind of NAN in, e.g., vanilla languages such as Python. The "issue with automatic differentiation" is really an issue with dividing by zero. Graph x(1./3.). It's slope is vertical at zero.
  - gugagore 7y ago
    I don't think this is right. An autodiff package will likely report +inf for the derivative of cuberoot at 0. The problem is really that there are a lot of ways the function can have infinite slope. The sqrt function also has infinite slope at 0. Dividing by zero isn't where the NaN comes from.

eli_gottlieb 7y ago

Autodiff only works because analytical calculus can prove that differentiation is a functor over functions.

jjtheblunt 7y ago
over continuous functions?
- eli_gottlieb 7y ago
  Well technically, over differentiable functions, but yes. Derivatives can be evaluated by decomposing functions, taking elementary derivatives, and recomposing the functions, in a way that integrals can't be.

nthompson 7y ago

Automatic differentiation only works on functions which are locally complex analytic. It fails on things that fall outside that model, but are still differentiable. Daubechies wavelets are a good example of this.

ChrisRackauckas 7y ago
Julia's systems can make use of ChainRules.jl which doesn't assume locally complex analytic, and instead uses its Wirtinger derivatives (df/dz and df/dzbar), and if complex analytic just uses the well-known simplification df/dzbar=0. This allows for non-analytic functions to be used, and you just need to supply all primitives in Wirtinger form. Julia has enough non-ML people using this stuff that complex numbers actually get used here :).

deleted 7y ago

[deleted]

petschge 7y ago

Kids in school say "why do I to learn X, I am never going to need it in real live". And then you end up with adults who don't believe that the earth is round, that vaccines are low-risk and extremely beneficial and think that the existence of snow means that there is no climate change and that homeopathy and essential oils are a valid alternative to modern medicine, etc.

adamnemecek 7y ago
This is completely unrelated but nice job that you put me in the same group as flat earthers. Because, yes, if you complain about current calculus curriculum, you are not just a flat earther but also LITERALLY HITLER. It's not about learning x, it's about the approaches you learn to solve x.