7 ms·

And for those of us just tuning in, what is an "autograd"? From skimming the article and some quick web searching, it looks like some sort of math function that

And for those of us just tuning in, what is an "autograd"? From skimming the article and some quick web searching, it looks like some sort of math function that gets used a lot in ML (and I assume other places) so there are implementations in a bunch of major libraries?

kragen 3y ago
reverse-mode automatic differentiation to compute the gradient of a scalar with respect to some large vector of inputs that it's computed from, which is critically important for the most popular and most efficient mathematical optimization algorithms such as adam you can't really implement it in a library in a normal programming language, because it has to introspect on how your program is computing everything, which is not something that a library can normally do; you have to implement it in a programming language, at least an embedded domain-specific language but in some cases something like a fortran compiler
- KeplerBoy 3y ago
  of course you can implement it in a library. You just have to make sure the users are using your array objects with appropriately overloaded operators so you can keep track of the computational graph. That's how pretty much how all python DL frameworks are built. And thats always why you can't call .numpy() on a torch tensor which requires a gradient. Doing so would break the computational graph.
  - kragen 3y ago
    that's why the comment you're replying to says 'at least an embedded domain-specific language', kepler boy
    - KeplerBoy 3y ago
      I still consider "you can't really implement it in a library in a normal programming language" as misleading.
      - stavros 3y ago
        I'll second this, as I consider Python a "normal programming language" and this is available as a library there.
        kragen 3y ago
        i could be wrong, but as far as i know there isn't an automatic differentiation library in python which you can use to compute the gradient of an arbitrary python function, just the gradient of an arbitrary function expressed in an embedded domain-specific language (usually one implemented by the library itself). kepler boy said specifically that this is true of pytorch but as far as i know it's true of all of them
        stavros 3y ago
        Maybe, I guess our difference is in what we mean by "DSL". I think we're disagreeing on whether "this library provides functions you can arrange your graph out of" is "DSL" or "normal language functionality".
        kragen 3y ago
        there's a fuzzy boundary between 'embedded dsl' and 'just a library' (similar to the problem of 'what is a framework'), but at the point that a library provides functions you can use to build your computational graph out of, it's clearly not just an automatic differentiation library; it's some kind of computational-graph-composing library which perhaps sports automatic differentiation as one of its features. and what i'm saying is that, if you want to write an automatic differentiation library in most languages, it's basically going to have to be a compiler that statically transforms your algorithm, whether it's a compiler for fortran or for an embedded dsl this is less true for forward-mode automatic differentiation, which you can do a pretty reasonable job of in python as just a dual-number class which you can pass to many, though not all, ordinary numerical computation functions. unfortunately forward-mode automatic differentiation is unusably slow for gradients with respect to large parameter vectors
        3y ago
  - tuukkah 3y ago
    That's what "embedded domain-specific language" means and what makes it different from other (unembedded) DSLs: > A DomainSpecificLanguage that is defined as a library for a generic "host" programming language. The embedded DSL inherits the generic language constructs of its host language https://wiki.c2.com/?EmbeddedDomainSpecificLanguage https://wiki.c2.com/?EmbeddedDomainSpecificLanguage
  - mlazos 3y ago
    He’s saying that PyTorch in essence is a DSL embedded in python which introspects itself to perform automatic differentiation. Libraries can’t normally peek at what ops are performed on variables in a given language, only the compiler/runtime for that language can. In the end it’s semantics - is an overloaded operator a DSL or native language construct? I’d argue it’s a DSL because its definition is in the host language.
    - credit_guy 3y ago
      You can implement autograd as a library. Just take a look at this https://github.com/sradc/SmallPebble https://github.com/sradc/SmallPebble The first line of the description is: > SmallPebble is a minimal automatic differentiation and deep learning library written from scratch in Python, using NumPy/CuPy. The author of this library wrote a blog post, which is in my opinion the best introduction to reverse mode automatic differentiation that exists: https://sidsite.com/posts/autodiff/ https://sidsite.com/posts/autodiff/
      - kragen 3y ago
        this is a nice blog post, but, two things: - it's very much taking the edsl approach - it's only implementing forward-mode
        credit_guy 3y ago
        It is reverse mode, not forward mode. As for the edsl, you can claim it is, but why? There is no parser, no syntax, nothing that would make any user think they are learning a new language. Would you call numpy an edsl ?
        kragen 3y ago
        the text says it's reverse mode, but it's wrong. what the code in the post implements is forward mode embedded dsls don't have parsers; that's what makes them embedded. numpy is solidly in the center of the embedded dsl concept
- HarHarVeryFunny 3y ago
  > you can't really implement it in a library in a normal programming language, because it has to introspect on how your program is computing everything AFAIK autograd generically refers to a computational graph's gradient being automatically derived from those of it's constituent computational nodes/operators. It's not limited to cases like PyTorch that presents itself as "differentiable programming" where the graph is implicitly built by code execution. The alternative to a differentiable language is something like the original Lua-based Torch, or TensorFlow's original non-eager mode, where the computational graph is explicitly built by library calls out of predefined nodes/operators, each of which knows it's own gradient.
  - kragen 3y ago
    agreed

kragen 3y ago

reverse-mode automatic differentiation to compute the gradient of a scalar with respect to some large vector of inputs that it's computed from, which is critically important for the most popular and most efficient mathematical optimization algorithms such as adam you can't really implement it in a library in a normal programming language, because it has to introspect on how your program is computing everything, which is not something that a library can normally do; you have to implement it in a programming language, at least an embedded domain-specific language but in some cases something like a fortran compiler

KeplerBoy 3y ago
of course you can implement it in a library. You just have to make sure the users are using your array objects with appropriately overloaded operators so you can keep track of the computational graph. That's how pretty much how all python DL frameworks are built. And thats always why you can't call .numpy() on a torch tensor which requires a gradient. Doing so would break the computational graph.
- kragen 3y ago
  that's why the comment you're replying to says 'at least an embedded domain-specific language', kepler boy
  - KeplerBoy 3y ago
    I still consider "you can't really implement it in a library in a normal programming language" as misleading.
    - stavros 3y ago
      I'll second this, as I consider Python a "normal programming language" and this is available as a library there.
      - kragen 3y ago
        i could be wrong, but as far as i know there isn't an automatic differentiation library in python which you can use to compute the gradient of an arbitrary python function, just the gradient of an arbitrary function expressed in an embedded domain-specific language (usually one implemented by the library itself). kepler boy said specifically that this is true of pytorch but as far as i know it's true of all of them
        stavros 3y ago
        Maybe, I guess our difference is in what we mean by "DSL". I think we're disagreeing on whether "this library provides functions you can arrange your graph out of" is "DSL" or "normal language functionality".
        kragen 3y ago
        there's a fuzzy boundary between 'embedded dsl' and 'just a library' (similar to the problem of 'what is a framework'), but at the point that a library provides functions you can use to build your computational graph out of, it's clearly not just an automatic differentiation library; it's some kind of computational-graph-composing library which perhaps sports automatic differentiation as one of its features. and what i'm saying is that, if you want to write an automatic differentiation library in most languages, it's basically going to have to be a compiler that statically transforms your algorithm, whether it's a compiler for fortran or for an embedded dsl this is less true for forward-mode automatic differentiation, which you can do a pretty reasonable job of in python as just a dual-number class which you can pass to many, though not all, ordinary numerical computation functions. unfortunately forward-mode automatic differentiation is unusably slow for gradients with respect to large parameter vectors
        3y ago
- tuukkah 3y ago
  That's what "embedded domain-specific language" means and what makes it different from other (unembedded) DSLs: > A DomainSpecificLanguage that is defined as a library for a generic "host" programming language. The embedded DSL inherits the generic language constructs of its host language https://wiki.c2.com/?EmbeddedDomainSpecificLanguage https://wiki.c2.com/?EmbeddedDomainSpecificLanguage
- mlazos 3y ago
  He’s saying that PyTorch in essence is a DSL embedded in python which introspects itself to perform automatic differentiation. Libraries can’t normally peek at what ops are performed on variables in a given language, only the compiler/runtime for that language can. In the end it’s semantics - is an overloaded operator a DSL or native language construct? I’d argue it’s a DSL because its definition is in the host language.
  - credit_guy 3y ago
    You can implement autograd as a library. Just take a look at this https://github.com/sradc/SmallPebble https://github.com/sradc/SmallPebble The first line of the description is: > SmallPebble is a minimal automatic differentiation and deep learning library written from scratch in Python, using NumPy/CuPy. The author of this library wrote a blog post, which is in my opinion the best introduction to reverse mode automatic differentiation that exists: https://sidsite.com/posts/autodiff/ https://sidsite.com/posts/autodiff/
    - kragen 3y ago
      this is a nice blog post, but, two things: - it's very much taking the edsl approach - it's only implementing forward-mode
      - credit_guy 3y ago
        It is reverse mode, not forward mode. As for the edsl, you can claim it is, but why? There is no parser, no syntax, nothing that would make any user think they are learning a new language. Would you call numpy an edsl ?
        kragen 3y ago
        the text says it's reverse mode, but it's wrong. what the code in the post implements is forward mode embedded dsls don't have parsers; that's what makes them embedded. numpy is solidly in the center of the embedded dsl concept
HarHarVeryFunny 3y ago
> you can't really implement it in a library in a normal programming language, because it has to introspect on how your program is computing everything AFAIK autograd generically refers to a computational graph's gradient being automatically derived from those of it's constituent computational nodes/operators. It's not limited to cases like PyTorch that presents itself as "differentiable programming" where the graph is implicitly built by code execution. The alternative to a differentiable language is something like the original Lua-based Torch, or TensorFlow's original non-eager mode, where the computational graph is explicitly built by library calls out of predefined nodes/operators, each of which knows it's own gradient.
- kragen 3y ago
  agreed