5 ms·

If you're surprised by this result because you've used kernel machines and didn't find them very good at generalization, keep in mind that this assumes a kernel

by tlb 6y ago

If you're surprised by this result because you've used kernel machines and didn't find them very good at generalization, keep in mind that this assumes a kernel function that accurately reflects similarity of input samples. Most work with kernel machines just uses Euclidean distance. For instance, in an image recognition model it would have to identify 2 images with dogs as more similar than an image with a dog and an image with a cat. With a sufficiently magical kernel function, indeed you can get great results with a kernel machine. But it's not so easy to write a kernel function for a domain like image processing, where shifts, scales, and small rotations shouldn't affect similarity much. Let alone for text processing, where it should recognize 2 sentences with similar meaning as similar.

yudlejoza 6y ago
I may or may not be surprised by the result, I'm definitely not surprised by the 'Thing A is thing B' in machine learning. Every tom, machine-learner, and harry, is an expert on proving to the whole world that thing-A is thing-B. The only problem is people don't hire them with a million dollars a year total compensation.
- memming 6y ago
  But the converse is not true in this case, so still interesting.
- yudlejoza 6y ago
  is an expert == is an "expert"
mywittyname 6y ago
>With a sufficiently magical kernel function, indeed you can get great results with a kernel machine. But it's not so easy to write a kernel function for a domain like image processing, where shifts, scales, and small rotations shouldn't affect similarity much. Let alone for text processing, where it should recognize 2 sentences with similar meaning as similar. I think the key issue at hand is that gradient descent is easier to train than a model using kernel functions. Someone could absolutely devise a mechanism for back propagation of errors with kernel functions, but at that point, it is basically a neural network.
- ced 6y ago
  Wouldn't that (similarity matters) be very useful information to encode in the NN though? Kinda like how you can do non-convolutional deep-learning on images, but it takes exponentially more training data than convolutional deep-learning does.
contravariant 6y ago
This may not have been your intention but I think you're underselling the utility of this result a bit. Yes you need a good kernel function to get good results but this paper literally provides a way to create such a kernel. With a way to turn a gradient descent into a roughly equivalent kernel machine you can turn gradient descent into an embedding, so rather than merely seemingly learning features you can recover the actual feature space. It's also interesting how the result heavily depends on the path taking during the gradient descent rather than merely the end result. This is somewhat counterintuitive when for the equivalent gradient descent only the end result is seemingly important.
- rq1 6y ago
  > It's also interesting how the result heavily depends on the path taking during the gradient descent rather than merely the end result. This is somewhat counterintuitive when for the equivalent gradient descent only the end result is seemingly important. I don’t understand. What do you mean here? I only (quickly) skimmed the article but it says that the limit is equal to a (not the) zero of your gradient. The result should be the same for every trajectory leading to the same zero. Put another way, it doesn’t depend on the path if it leads to the same zero. I would say here that not all zeros are equal though. :)
- leereeves 6y ago
  > It's also interesting how the result heavily depends on the path taking during the gradient descent rather than merely the end result. After reading just the linked article, not the original paper, my interpretation is that the kernel isn't calculated during the gradient descent process, but from the final model after training by gradient descent. If that's right, it's not path-dependent.
  - l33tman 6y ago
    No, the kernel is a mixing of gradients from the training. The proof depends completely on the particular form of optimization, in this case gradient descent in the limit of infinitesimal "timestepping", and suggests that gradient descent optimization doesn't cleverly pick good internal representations of the data.
freeone3000 6y ago
Well, we have the sufficiently magical kernel function from the gradient descent process -- seems like we could save a lot of computation by extracting out that function and running it as an SVM rather than holding it independent.
- davnn 6y ago
  Interesting idea! Is there any research in this direction?
thwoeir234234 6y ago
> If you're surprised by this result because you've used kernel machines and didn't find them very good at generalization, This statement is a bit strange, because most of the theoretical work around NTK assumes that the tangent kernel doesn't change sufficiently during training. This seemed to make this closer to "extreme machines" than deep learning. There was a paper from Francis Bach's group which seemed to be saying as much, but that criticism was no match for the meme-fication of this line of research by over-enthusiastic amateurs.

by tlb 6y ago

yudlejoza 6y ago

I may or may not be surprised by the result, I'm definitely not surprised by the 'Thing A is thing B' in machine learning. Every tom, machine-learner, and harry, is an expert on proving to the whole world that thing-A is thing-B. The only problem is people don't hire them with a million dollars a year total compensation.

memming 6y ago
But the converse is not true in this case, so still interesting.
yudlejoza 6y ago
is an expert == is an "expert"

mywittyname 6y ago

>With a sufficiently magical kernel function, indeed you can get great results with a kernel machine. But it's not so easy to write a kernel function for a domain like image processing, where shifts, scales, and small rotations shouldn't affect similarity much. Let alone for text processing, where it should recognize 2 sentences with similar meaning as similar. I think the key issue at hand is that gradient descent is easier to train than a model using kernel functions. Someone could absolutely devise a mechanism for back propagation of errors with kernel functions, but at that point, it is basically a neural network.

ced 6y ago
Wouldn't that (similarity matters) be very useful information to encode in the NN though? Kinda like how you can do non-convolutional deep-learning on images, but it takes exponentially more training data than convolutional deep-learning does.

contravariant 6y ago

This may not have been your intention but I think you're underselling the utility of this result a bit. Yes you need a good kernel function to get good results but this paper literally provides a way to create such a kernel. With a way to turn a gradient descent into a roughly equivalent kernel machine you can turn gradient descent into an embedding, so rather than merely seemingly learning features you can recover the actual feature space. It's also interesting how the result heavily depends on the path taking during the gradient descent rather than merely the end result. This is somewhat counterintuitive when for the equivalent gradient descent only the end result is seemingly important.

rq1 6y ago
> It's also interesting how the result heavily depends on the path taking during the gradient descent rather than merely the end result. This is somewhat counterintuitive when for the equivalent gradient descent only the end result is seemingly important. I don’t understand. What do you mean here? I only (quickly) skimmed the article but it says that the limit is equal to a (not the) zero of your gradient. The result should be the same for every trajectory leading to the same zero. Put another way, it doesn’t depend on the path if it leads to the same zero. I would say here that not all zeros are equal though. :)
leereeves 6y ago
> It's also interesting how the result heavily depends on the path taking during the gradient descent rather than merely the end result. After reading just the linked article, not the original paper, my interpretation is that the kernel isn't calculated during the gradient descent process, but from the final model after training by gradient descent. If that's right, it's not path-dependent.
- l33tman 6y ago
  No, the kernel is a mixing of gradients from the training. The proof depends completely on the particular form of optimization, in this case gradient descent in the limit of infinitesimal "timestepping", and suggests that gradient descent optimization doesn't cleverly pick good internal representations of the data.

freeone3000 6y ago

Well, we have the sufficiently magical kernel function from the gradient descent process -- seems like we could save a lot of computation by extracting out that function and running it as an SVM rather than holding it independent.

davnn 6y ago
Interesting idea! Is there any research in this direction?

thwoeir234234 6y ago

> If you're surprised by this result because you've used kernel machines and didn't find them very good at generalization, This statement is a bit strange, because most of the theoretical work around NTK assumes that the tangent kernel doesn't change sufficiently during training. This seemed to make this closer to "extreme machines" than deep learning. There was a paper from Francis Bach's group which seemed to be saying as much, but that criticism was no match for the meme-fication of this line of research by over-enthusiastic amateurs.