5 ms·

This may not have been your intention but I think you're underselling the utility of this result a bit. Yes you need a good kernel function to get good results

This may not have been your intention but I think you're underselling the utility of this result a bit. Yes you need a good kernel function to get good results but this paper literally provides a way to create such a kernel. With a way to turn a gradient descent into a roughly equivalent kernel machine you can turn gradient descent into an embedding, so rather than merely seemingly learning features you can recover the actual feature space. It's also interesting how the result heavily depends on the path taking during the gradient descent rather than merely the end result. This is somewhat counterintuitive when for the equivalent gradient descent only the end result is seemingly important.

rq1 6y ago
> It's also interesting how the result heavily depends on the path taking during the gradient descent rather than merely the end result. This is somewhat counterintuitive when for the equivalent gradient descent only the end result is seemingly important. I don’t understand. What do you mean here? I only (quickly) skimmed the article but it says that the limit is equal to a (not the) zero of your gradient. The result should be the same for every trajectory leading to the same zero. Put another way, it doesn’t depend on the path if it leads to the same zero. I would say here that not all zeros are equal though. :)
leereeves 6y ago
> It's also interesting how the result heavily depends on the path taking during the gradient descent rather than merely the end result. After reading just the linked article, not the original paper, my interpretation is that the kernel isn't calculated during the gradient descent process, but from the final model after training by gradient descent. If that's right, it's not path-dependent.
- l33tman 6y ago
  No, the kernel is a mixing of gradients from the training. The proof depends completely on the particular form of optimization, in this case gradient descent in the limit of infinitesimal "timestepping", and suggests that gradient descent optimization doesn't cleverly pick good internal representations of the data.
  - leereeves 6y ago
    > suggests that gradient descent optimization doesn't cleverly pick good internal representations of the data. How so? The existence of equivalent kernels doesn't imply they aren't clever or good.
- contravariant 6y ago
  The final kernel is an integral taken over the path of the parameters during the gradient descent. Basically what they're doing is just y = y0 + \int dy/dt dt and then rephrasing this equation as a kernel machine. Which I guess should also work for other forms of learning where you slowly change the parameters.

rq1 6y ago

> It's also interesting how the result heavily depends on the path taking during the gradient descent rather than merely the end result. This is somewhat counterintuitive when for the equivalent gradient descent only the end result is seemingly important. I don’t understand. What do you mean here? I only (quickly) skimmed the article but it says that the limit is equal to a (not the) zero of your gradient. The result should be the same for every trajectory leading to the same zero. Put another way, it doesn’t depend on the path if it leads to the same zero. I would say here that not all zeros are equal though. :)

leereeves 6y ago

> It's also interesting how the result heavily depends on the path taking during the gradient descent rather than merely the end result. After reading just the linked article, not the original paper, my interpretation is that the kernel isn't calculated during the gradient descent process, but from the final model after training by gradient descent. If that's right, it's not path-dependent.

l33tman 6y ago
No, the kernel is a mixing of gradients from the training. The proof depends completely on the particular form of optimization, in this case gradient descent in the limit of infinitesimal "timestepping", and suggests that gradient descent optimization doesn't cleverly pick good internal representations of the data.
- leereeves 6y ago
  > suggests that gradient descent optimization doesn't cleverly pick good internal representations of the data. How so? The existence of equivalent kernels doesn't imply they aren't clever or good.
contravariant 6y ago
The final kernel is an integral taken over the path of the parameters during the gradient descent. Basically what they're doing is just y = y0 + \int dy/dt dt and then rephrasing this equation as a kernel machine. Which I guess should also work for other forms of learning where you slowly change the parameters.