3 ms·

The idea behind the term "differential programming" is that many traditional autodiff systems force you into a highly restricted subset of the language or DSL (

The idea behind the term "differential programming" is that many traditional autodiff systems force you into a highly restricted subset of the language or DSL (e.g. no loops, no if statements etc). Differential programming is a term used to describe systems that let you take derivatives of your entire source code. This lets you do things like AD through a simulation or optimization algorithm which can be really powerful for applying ML techniques to domains that are less typically suited to ML approaches (circuits, physics, differential equations etc)

bmitc 4y ago
But I would wager that those are poor implementations of automatic differentiation, as the property that automatic differentiation works with for loops, if statements, etc. is inherent to automatic differentiation. So differential programming seems like automatic differentiation just implemented properly. I've written some simple forward-mode automatic differentiation implementations in a few languages, and it's akin to just using a library. It doesn't seem like a paradigm to me. Reverse-mode automatic differentiation is basically backpropagation. So again, ML seems to just like to rename things. I studied mathematics and have recently been trying to learn some ML. It's pretty annoying that a lot of mathematical terms seem to have been renamed or coopted for something else.
- oddity 4y ago
  Autodiff does not work with for loops or if statements. The current solutions effectively pick a few promising traces through the program and then assume that nothing else exists. To handle it more elegantly (for things like preserving equational reasoning or avoiding exponential blowup) you need to address it at the level of language semantics.
  - bmitc 4y ago
    > Autodiff does not work with for loops or if statements. Is that necessarily true? Here is an incomplete automatic differentiation implementation that handles if statements just fine in a function definition. Unless you mean something else. type Dual = {Real: float; Epsilon: float} with static member (~-) (x: Dual) = {Real = -x.Real; Epsilon = -x.Epsilon} static member (+) (x: Dual, y: Dual) = { Real = x.Real + y.Real Epsilon = x.Epsilon + y.Epsilon } static member (+) (x: Dual, c: float) = {x with Real = x.Real + c} static member (+) (c: float, y: Dual) = {y with Real = c + y.Real} static member (-) (x: Dual, y: Dual) = x + (-y) static member (-) (x: Dual, c: float) = x + (-c) static member (-) (c: float, y: Dual) = c + (-y) static member (*) (x: Dual, y: Dual) = { Real = x.Real * y.Real Epsilon = x.Real * y.Epsilon + x.Epsilon * y.Real } static member (*) (c: float, y: Dual) = {Real = c; Epsilon = 0} * y static member (*) (x: Dual, c: float) = x * {Real = c; Epsilon = 0} let dcos (x: Dual) = {Real = cos x.Real; Epsilon = -(sin x.Real) * x.Epsilon} let dsin (x: Dual) = {Real = sin x.Real; Epsilon = (cos x.Real) * x.Epsilon} let differentiate (f: Dual -> Dual) a = let x = f {Real = a; Epsilon = 1.0} x.Epsilon let testFunction (x: Dual) = if x.Real < 0.0 then dcos x else dsin (x*x - 3.0*x) Using that gives: > differentiate testFunction -1.0 0.8414709848078965 > differentiate testFunction 1.0 0.4161468365471424 > differentiate testFunction 0.0 -3.0 Now, of course, one needs to be careful interpreting the result at a = 0.0. That's because the testFunction is not differentiable at that point due to a jump discontinuity there, but we still get a value back. But as far as I know, this is simply an issue with automatic differentiation in that it only correctly tells you what the derivative is if it exists at the given point.
    - data_maan 4y ago
      @bmitc: Reading your replies (some of which seem to have been written the same time I wrote mine), it seems we are on the same page; I'm also a mathematician and I also have some qualms with how people invent new names for automatic differentiation :) I had a look at your bio and couldn't find any email address. Would you perhaps be interested in having a longer, scientific discussion about AD?
    - oddity 4y ago
      This "discretizes then differentiates" to borrow terminology from [1] which is one of the more accessible presentations and papers. The program might evaluate correctly, but equational reasoning (like you might want for any kind of automated optimizations) is broken. In a toy example like this where you're doing everything manually then you probably don't care, but for larger systems, it gets tiring to do the mental equivalent of assembly programming. [1] https://people.csail.mit.edu/sbangaru/projects/teg-2021/ https://people.csail.mit.edu/sbangaru/projects/teg-2021/
      - data_maan 4y ago
        > In a toy example like this where you're doing everything manually then you probably don't care, but for larger systems, it gets tiring to do the mental equivalent of assembly programming. But by this argument (which sounds plausible to me) you have defeated your previous claim that differential programming is really a new paradigm, as it seems you have adopted what bmitc wrote earlier, that differential programming is not a new paradigm but "seems like automatic differentiation just implemented properly".
        oddity 4y ago
        There's no contradiction: autodiff is a method of implementing differentiable programming. In this example, it is implemented as a type that handles a trace of a program, but everything else is left to the programmer. This is a problem because most of the code I would want to write is not a single trace! Analogously, I could write a program in C that does message sends and organizes code in a design pattern called "objects" and "classes". Incredibly painful, but workable sometimes. Some people even call it "object oriented C" and go on to create a library to handle it like [1]. Is object orientation not a paradigm because I've implemented a core piece as a library? No, because that misses the intangible part of what makes a paradigm a paradigm: I structured my code this way, for a reason. In OOP, that reason is the compartmentalization of concerns. The underlying OOP mechanism gives me a way to reason about composition and substitution of components to minimize how much I have to reason about when writing code. Similarly, in differentiable programming, the differentiability of all things gives me a way to reason about the smooth substitution of things because it more easily lets me reason about how the machine writes code. [1] https://en.wikipedia.org/wiki/GObject https://en.wikipedia.org/wiki/GObject
  - QuackingTheQ 4y ago
    Julia's AD is compatible with control flow. They have their own issues, but Zygote + ChainRules actually work pretty well
data_maan 4y ago
Mathematician here. > Differential programming is a term used to describe systems that let you take derivatives of your entire source code. I think the way this is stated is incorrect. What would the derivative of a program that outputs the reverse the input string be? It would be meaningless. It seems rather to be the case that it describes systems that let you take derivatives of your entire source code where that source code describes a numerical function. Perhaps it is obvious to state this, but I keep seeing differentiable programming being described as "taking a derivative of the source code" and it seems annoyingly imprecise. > This lets you do things like AD through a simulation or optimization algorithm This is also, taken literally, not a correct statement. It's not the algorithm which you want to differentiate, but a hard-to-differentiate function within the algorithm (which is always a gradient-descent type algorithm, as it otherwise makes no sense to consider derivatives; there is the entire domain of derivative-free optimization BTW). So, in case of (stochastic) gradient descent to train a neural network, that would be the neural network that you want to use AD on, not the gradient descent algorithm. Obviously.
- radarsat1 4y ago
  > So, in case of (stochastic) gradient descent to train a neural network, that would be the neural network that you want to use AD on, not the gradient descent algorithm. Obviously. The specific proposal of differential programming as a paradigm that goes beyond simple applications of gradient descent for optimising NNs is exactly to apply gradient descent to optimise learning (and other) algorithms themselves. This may be to select hyperparameters, or to select among a family of algorithms, or to include differentiable constraints in the solver, etc. I wonder, are you familiar with now classic paper, Learning to learn by gradient descent by gradient descent? [0] https://arxiv.org/abs/1606.04474 https://arxiv.org/abs/1606.04474
  - selestify 4y ago
    Wow, it’s interesting that a paper from 2016 is already classic. This field moves fast!
    - deleted 4y ago

bmitc 4y ago

But I would wager that those are poor implementations of automatic differentiation, as the property that automatic differentiation works with for loops, if statements, etc. is inherent to automatic differentiation. So differential programming seems like automatic differentiation just implemented properly. I've written some simple forward-mode automatic differentiation implementations in a few languages, and it's akin to just using a library. It doesn't seem like a paradigm to me. Reverse-mode automatic differentiation is basically backpropagation. So again, ML seems to just like to rename things. I studied mathematics and have recently been trying to learn some ML. It's pretty annoying that a lot of mathematical terms seem to have been renamed or coopted for something else.

oddity 4y ago
Autodiff does not work with for loops or if statements. The current solutions effectively pick a few promising traces through the program and then assume that nothing else exists. To handle it more elegantly (for things like preserving equational reasoning or avoiding exponential blowup) you need to address it at the level of language semantics.
- bmitc 4y ago
  > Autodiff does not work with for loops or if statements. Is that necessarily true? Here is an incomplete automatic differentiation implementation that handles if statements just fine in a function definition. Unless you mean something else. type Dual = {Real: float; Epsilon: float} with static member (~-) (x: Dual) = {Real = -x.Real; Epsilon = -x.Epsilon} static member (+) (x: Dual, y: Dual) = { Real = x.Real + y.Real Epsilon = x.Epsilon + y.Epsilon } static member (+) (x: Dual, c: float) = {x with Real = x.Real + c} static member (+) (c: float, y: Dual) = {y with Real = c + y.Real} static member (-) (x: Dual, y: Dual) = x + (-y) static member (-) (x: Dual, c: float) = x + (-c) static member (-) (c: float, y: Dual) = c + (-y) static member (*) (x: Dual, y: Dual) = { Real = x.Real * y.Real Epsilon = x.Real * y.Epsilon + x.Epsilon * y.Real } static member (*) (c: float, y: Dual) = {Real = c; Epsilon = 0} * y static member (*) (x: Dual, c: float) = x * {Real = c; Epsilon = 0} let dcos (x: Dual) = {Real = cos x.Real; Epsilon = -(sin x.Real) * x.Epsilon} let dsin (x: Dual) = {Real = sin x.Real; Epsilon = (cos x.Real) * x.Epsilon} let differentiate (f: Dual -> Dual) a = let x = f {Real = a; Epsilon = 1.0} x.Epsilon let testFunction (x: Dual) = if x.Real < 0.0 then dcos x else dsin (x*x - 3.0*x) Using that gives: > differentiate testFunction -1.0 0.8414709848078965 > differentiate testFunction 1.0 0.4161468365471424 > differentiate testFunction 0.0 -3.0 Now, of course, one needs to be careful interpreting the result at a = 0.0. That's because the testFunction is not differentiable at that point due to a jump discontinuity there, but we still get a value back. But as far as I know, this is simply an issue with automatic differentiation in that it only correctly tells you what the derivative is if it exists at the given point.
  - data_maan 4y ago
    @bmitc: Reading your replies (some of which seem to have been written the same time I wrote mine), it seems we are on the same page; I'm also a mathematician and I also have some qualms with how people invent new names for automatic differentiation :) I had a look at your bio and couldn't find any email address. Would you perhaps be interested in having a longer, scientific discussion about AD?
  - oddity 4y ago
    This "discretizes then differentiates" to borrow terminology from [1] which is one of the more accessible presentations and papers. The program might evaluate correctly, but equational reasoning (like you might want for any kind of automated optimizations) is broken. In a toy example like this where you're doing everything manually then you probably don't care, but for larger systems, it gets tiring to do the mental equivalent of assembly programming. [1] https://people.csail.mit.edu/sbangaru/projects/teg-2021/ https://people.csail.mit.edu/sbangaru/projects/teg-2021/
    - data_maan 4y ago
      > In a toy example like this where you're doing everything manually then you probably don't care, but for larger systems, it gets tiring to do the mental equivalent of assembly programming. But by this argument (which sounds plausible to me) you have defeated your previous claim that differential programming is really a new paradigm, as it seems you have adopted what bmitc wrote earlier, that differential programming is not a new paradigm but "seems like automatic differentiation just implemented properly".
      - oddity 4y ago
        There's no contradiction: autodiff is a method of implementing differentiable programming. In this example, it is implemented as a type that handles a trace of a program, but everything else is left to the programmer. This is a problem because most of the code I would want to write is not a single trace! Analogously, I could write a program in C that does message sends and organizes code in a design pattern called "objects" and "classes". Incredibly painful, but workable sometimes. Some people even call it "object oriented C" and go on to create a library to handle it like [1]. Is object orientation not a paradigm because I've implemented a core piece as a library? No, because that misses the intangible part of what makes a paradigm a paradigm: I structured my code this way, for a reason. In OOP, that reason is the compartmentalization of concerns. The underlying OOP mechanism gives me a way to reason about composition and substitution of components to minimize how much I have to reason about when writing code. Similarly, in differentiable programming, the differentiability of all things gives me a way to reason about the smooth substitution of things because it more easily lets me reason about how the machine writes code. [1] https://en.wikipedia.org/wiki/GObject https://en.wikipedia.org/wiki/GObject
- QuackingTheQ 4y ago
  Julia's AD is compatible with control flow. They have their own issues, but Zygote + ChainRules actually work pretty well

data_maan 4y ago

Mathematician here. > Differential programming is a term used to describe systems that let you take derivatives of your entire source code. I think the way this is stated is incorrect. What would the derivative of a program that outputs the reverse the input string be? It would be meaningless. It seems rather to be the case that it describes systems that let you take derivatives of your entire source code where that source code describes a numerical function. Perhaps it is obvious to state this, but I keep seeing differentiable programming being described as "taking a derivative of the source code" and it seems annoyingly imprecise. > This lets you do things like AD through a simulation or optimization algorithm This is also, taken literally, not a correct statement. It's not the algorithm which you want to differentiate, but a hard-to-differentiate function within the algorithm (which is always a gradient-descent type algorithm, as it otherwise makes no sense to consider derivatives; there is the entire domain of derivative-free optimization BTW). So, in case of (stochastic) gradient descent to train a neural network, that would be the neural network that you want to use AD on, not the gradient descent algorithm. Obviously.

radarsat1 4y ago
> So, in case of (stochastic) gradient descent to train a neural network, that would be the neural network that you want to use AD on, not the gradient descent algorithm. Obviously. The specific proposal of differential programming as a paradigm that goes beyond simple applications of gradient descent for optimising NNs is exactly to apply gradient descent to optimise learning (and other) algorithms themselves. This may be to select hyperparameters, or to select among a family of algorithms, or to include differentiable constraints in the solver, etc. I wonder, are you familiar with now classic paper, Learning to learn by gradient descent by gradient descent? [0] https://arxiv.org/abs/1606.04474 https://arxiv.org/abs/1606.04474
- selestify 4y ago
  Wow, it’s interesting that a paper from 2016 is already classic. This field moves fast!
  - deleted 4y ago