3 ms·

I still consider "you can't really implement it in a library in a normal programming language" as misleading.

• stavros 3y ago
  I'll second this, as I consider Python a "normal programming language" and this is available as a library there.
  • kragen 3y ago
    i could be wrong, but as far as i know there isn't an automatic differentiation library in python which you can use to compute the gradient of an arbitrary python function, just the gradient of an arbitrary function expressed in an embedded domain-specific language (usually one implemented by the library itself). kepler boy said specifically that this is true of pytorch but as far as i know it's true of all of them
    • stavros 3y ago
      Maybe, I guess our difference is in what we mean by "DSL". I think we're disagreeing on whether "this library provides functions you can arrange your graph out of" is "DSL" or "normal language functionality".
      • kragen 3y ago
        there's a fuzzy boundary between 'embedded dsl' and 'just a library' (similar to the problem of 'what is a framework'), but at the point that a library provides functions you can use to build your computational graph out of, it's clearly not just an automatic differentiation library; it's some kind of computational-graph-composing library which perhaps sports automatic differentiation as one of its features. and what i'm saying is that, if you want to write an automatic differentiation library in most languages, it's basically going to have to be a compiler that statically transforms your algorithm, whether it's a compiler for fortran or for an embedded dsl this is less true for forward-mode automatic differentiation, which you can do a pretty reasonable job of in python as just a dual-number class which you can pass to many, though not all, ordinary numerical computation functions. unfortunately forward-mode automatic differentiation is unusably slow for gradients with respect to large parameter vectors
      • mlazos 3y ago
        It’s a DSL by definition because a function provided by an external library cannot be a language primitive in the host language, or else it wouldn’t need to be an external library! Unless we’re talking about a LISP but we’re not.
        • stavros 3y ago
          All libraries are DSLs, then? The initial claim was: > you can't really implement it in a library in a normal programming [...] you have to implement it in a programming language, at least an embedded domain-specific language So which is it?
          • mlazos 3y ago
            No lol not all libraries are DSLs, we’re talking specifically about libraries which provide functions allowing you to construct a computational graph. These libraries are DSLs, not host language primitives.
            • stavros 3y ago
              What's the difference between a library function and a DSL library? What's a library function that's also a host language primitive?
              • mlazos 3y ago
                A DSL lib is a subset of “all possible function libs” and there is no such thing as a function library which is a host language primitive by definition. You can’t be a primitive of a language if you’re implemented in a library for that language.
            • kragen 3y ago
              regexps, html, and css2 are also dsls despite not providing functions allowing you to construct a computational graph, but i agree that at the point where you have a graph of computations you have accidentally a programming language