3 ms·

Mathematician here. > Differential programming is a term used to describe systems that let you take derivatives of your entire source code. I think the way th

Mathematician here. > Differential programming is a term used to describe systems that let you take derivatives of your entire source code. I think the way this is stated is incorrect. What would the derivative of a program that outputs the reverse the input string be? It would be meaningless. It seems rather to be the case that it describes systems that let you take derivatives of your entire source code where that source code describes a numerical function. Perhaps it is obvious to state this, but I keep seeing differentiable programming being described as "taking a derivative of the source code" and it seems annoyingly imprecise. > This lets you do things like AD through a simulation or optimization algorithm This is also, taken literally, not a correct statement. It's not the algorithm which you want to differentiate, but a hard-to-differentiate function within the algorithm (which is always a gradient-descent type algorithm, as it otherwise makes no sense to consider derivatives; there is the entire domain of derivative-free optimization BTW). So, in case of (stochastic) gradient descent to train a neural network, that would be the neural network that you want to use AD on, not the gradient descent algorithm. Obviously.

radarsat1 4y ago
> So, in case of (stochastic) gradient descent to train a neural network, that would be the neural network that you want to use AD on, not the gradient descent algorithm. Obviously. The specific proposal of differential programming as a paradigm that goes beyond simple applications of gradient descent for optimising NNs is exactly to apply gradient descent to optimise learning (and other) algorithms themselves. This may be to select hyperparameters, or to select among a family of algorithms, or to include differentiable constraints in the solver, etc. I wonder, are you familiar with now classic paper, Learning to learn by gradient descent by gradient descent? [0] https://arxiv.org/abs/1606.04474 https://arxiv.org/abs/1606.04474
- selestify 4y ago
  Wow, it’s interesting that a paper from 2016 is already classic. This field moves fast!
  - deleted 4y ago
    [deleted]
  - radarsat1 4y ago
    Well, I'd say it's a "classic" within the field of differentiable programming, which is a new field, so... everything is relative.
- HarHarVeryFunny 4y ago
  I don't really see the distinction. In the basic case of using gradients to optimize a NN, what's really being optimized are the parameters of a function expressing the NN error. We're just following the gradient downhill to find whatever minimum of this function the gradient leads us to (hopefully the global minimum, as is likely to be the case with lots of parameters). In this "learning to learn" paper the optimization function (used to update the gradients at each step of the gradient descent algorithm) has been parameterized, and we're instead using gradient descent to find a minimum of that parameterized function. So, yes, differentiable programming isn't limited to minimizing the losses of neural nets, but as the grandparent poster pointed out, the whole concept obviously only applies to parameterized programs calculating some numerical output that we want to miminize.

radarsat1 4y ago

> So, in case of (stochastic) gradient descent to train a neural network, that would be the neural network that you want to use AD on, not the gradient descent algorithm. Obviously. The specific proposal of differential programming as a paradigm that goes beyond simple applications of gradient descent for optimising NNs is exactly to apply gradient descent to optimise learning (and other) algorithms themselves. This may be to select hyperparameters, or to select among a family of algorithms, or to include differentiable constraints in the solver, etc. I wonder, are you familiar with now classic paper, Learning to learn by gradient descent by gradient descent? [0] https://arxiv.org/abs/1606.04474 https://arxiv.org/abs/1606.04474

selestify 4y ago
Wow, it’s interesting that a paper from 2016 is already classic. This field moves fast!
- deleted 4y ago
  [deleted]
- radarsat1 4y ago
  Well, I'd say it's a "classic" within the field of differentiable programming, which is a new field, so... everything is relative.
HarHarVeryFunny 4y ago
I don't really see the distinction. In the basic case of using gradients to optimize a NN, what's really being optimized are the parameters of a function expressing the NN error. We're just following the gradient downhill to find whatever minimum of this function the gradient leads us to (hopefully the global minimum, as is likely to be the case with lots of parameters). In this "learning to learn" paper the optimization function (used to update the gradients at each step of the gradient descent algorithm) has been parameterized, and we're instead using gradient descent to find a minimum of that parameterized function. So, yes, differentiable programming isn't limited to minimizing the losses of neural nets, but as the grandparent poster pointed out, the whole concept obviously only applies to parameterized programs calculating some numerical output that we want to miminize.