5 ms·

Oh, much simpler than that - simply that you can pose the problem you are trying to solve in a way that the derivative can be calculated. If you can, you can b

by Fede_V 10y ago

Oh, much simpler than that - simply that you can pose the problem you are trying to solve in a way that the derivative can be calculated. If you can, you can bring about the whole deep learning machinery to bear on it.

kabouseng 10y ago
Just to be clear here, your meaning is that the reward function should be differentiable so there is a slope to climb to a maxima (local or gobal). - edit used the wrong word, used derivative meant differentiable.
- habitue 10y ago
  He's talking about the operations needing to be differentiable. So like multiplication, addition etc are differentiable. tanh, sigmoid functions are differentiable. You need this because back propagation is an application of the chain rule, so each constituent operation needs to be differentiable. A lot of interesting advances have come from making operations that seems like they aren't differentiable, into differentiable operations that can be trained by backpropagation. See LSTMs and neural turing machines
azakai 10y ago
Being able to calculate the derivative is very useful (the backprop rule, etc.). However it isn't actually necessary here, since you can approximate the slope by just sampling points nearly. Strict differentiability isn't the crucial property here, but it does make things faster. What is, I think, is that the error function is continuous and there is a monotonic path of improvement from most of the space to good solutions in it. Then you can just descend in the error function until you reach it. We don't really understand why that happens in deep learning, that is, why the error functions of deep neural networks tend to not have complex shapes with local minima that you can get stuck in. This is currently being investigated. This also isn't a problem limited to deep learning. In evolution, the same is true - we are limited to small local improvements, but over large amounts of time nature arrives at amazingly efficient solutions. That suggests that the fitness function space over genomes is also, to some extent, continuous and allows monotonic improvement. Again, we don't really know why, yet.
- Fede_V 10y ago
  Finite difference approximations to the derivative are not practical at all when doing deep learning. Derivative using backpropagations can be calculated at the same speed as the forward pass - while derivative approximated using finite differences require 2 passes PER parameter, and run into numerous numerical issues with accuracy. It's an absolute non-starter.
  - azakai 10y ago
    You don't actually need to calculate the derivative accurately. It's enough to sample several points near the current one and continue the search to the best of those. The point here is that we are descending in a continuous and monotonic loss function. You don't need an accurate derivative or even an estimate of the derivative to do so, although it definitely helps a great deal.
    - nightski 10y ago
      Except that you are talking about the derivative with respect to each weight. In a deep neural network, you'd need several sample points for each weight in order to compute dC/dw. That is a LOT of work for a single step. The only way to do this in a timely fashion is via backprop, which requires a derivative.
- TumTumTree 10y ago
  is there a term or subject area where one could learn more about the investigations underway on error functions that tend to not have complex shapes with local minima to get stuck in?
  - kirill5pol 10y ago
    It's called convex analysis, here's a course on the topic: https://lagunita.stanford.edu/courses/Engineering/CVX101/Winter2014/about https://lagunita.stanford.edu/courses/Engineering/CVX101/Win...

by Fede_V 10y ago

kabouseng 10y ago

Just to be clear here, your meaning is that the reward function should be differentiable so there is a slope to climb to a maxima (local or gobal). - edit used the wrong word, used derivative meant differentiable.

habitue 10y ago
He's talking about the operations needing to be differentiable. So like multiplication, addition etc are differentiable. tanh, sigmoid functions are differentiable. You need this because back propagation is an application of the chain rule, so each constituent operation needs to be differentiable. A lot of interesting advances have come from making operations that seems like they aren't differentiable, into differentiable operations that can be trained by backpropagation. See LSTMs and neural turing machines

azakai 10y ago

Being able to calculate the derivative is very useful (the backprop rule, etc.). However it isn't actually necessary here, since you can approximate the slope by just sampling points nearly. Strict differentiability isn't the crucial property here, but it does make things faster. What is, I think, is that the error function is continuous and there is a monotonic path of improvement from most of the space to good solutions in it. Then you can just descend in the error function until you reach it. We don't really understand why that happens in deep learning, that is, why the error functions of deep neural networks tend to not have complex shapes with local minima that you can get stuck in. This is currently being investigated. This also isn't a problem limited to deep learning. In evolution, the same is true - we are limited to small local improvements, but over large amounts of time nature arrives at amazingly efficient solutions. That suggests that the fitness function space over genomes is also, to some extent, continuous and allows monotonic improvement. Again, we don't really know why, yet.

Fede_V 10y ago
Finite difference approximations to the derivative are not practical at all when doing deep learning. Derivative using backpropagations can be calculated at the same speed as the forward pass - while derivative approximated using finite differences require 2 passes PER parameter, and run into numerous numerical issues with accuracy. It's an absolute non-starter.
- azakai 10y ago
  You don't actually need to calculate the derivative accurately. It's enough to sample several points near the current one and continue the search to the best of those. The point here is that we are descending in a continuous and monotonic loss function. You don't need an accurate derivative or even an estimate of the derivative to do so, although it definitely helps a great deal.
  - nightski 10y ago
    Except that you are talking about the derivative with respect to each weight. In a deep neural network, you'd need several sample points for each weight in order to compute dC/dw. That is a LOT of work for a single step. The only way to do this in a timely fashion is via backprop, which requires a derivative.
TumTumTree 10y ago
is there a term or subject area where one could learn more about the investigations underway on error functions that tend to not have complex shapes with local minima to get stuck in?
- kirill5pol 10y ago
  It's called convex analysis, here's a course on the topic: https://lagunita.stanford.edu/courses/Engineering/CVX101/Winter2014/about https://lagunita.stanford.edu/courses/Engineering/CVX101/Win...