6 ms·
Gradient Descent Models Are Kernel Machines
- throwawaysea 6y ago> This result makes it very clear that without regularity imposed by the ground truth mechanism which generates the actual data (e.g., some natural process), a neural net is unlikely to perform well on an example which deviates strongly (as defined by the kernel) from all training examples. Is this another way of saying that neural networks are just another statistical estimation method and not a path to general artificial intelligence? Or is it saying that problems like self-driving cars are not suitable to the current state of the art for AI since we have to ensure that reality doesn't deviate from training examples? Or both? I'd love to understand the "real life" implications of this finding better.
- phreeza 6y agoIs it not possible to achieve AGI with a good statistical estimation method?
- viraptor 6y agoDepends if you think "a human unlikely to perform well on an example which deviates strongly (as defined by experience) from all training examples." is true.
- fallingfrog 6y agoA human can learn from a single training sample. I can show my 2 year old a cartoon picture of an unfamiliar animal- say, an anteater- and then show him a page of drawings by a different artist in a different style, of several animals, in various poses, and he can immediately point to the anteater. Think about that: a single example, crudely drawn. You and I could do that too. Human beings are masters of learning from training sets of n=1. I’m not sure why humans are so good at that. I feel like it might be that by spending the first couple years of life picking things up and rotating them and putting them in our mouths, we have some deeper understanding that images of an animal like an anteater are not really two dimensional but are supposed to represent volume and form. That’s my guess. A neural net trained to recognize 2d images will not have that.
- viraptor 6y agoI don't think that's a fair comparison. It's not 1 sample. It's 1 sample + existing training in segmentation + existing model for expected animal build + existing model for 2d/3d extrapolation from the drawing + pre-trained capacity for reinforcement learning, etc. If you start slightly before 2yo, you may find that after they see a cow they will call every large animal a cow for weeks/months - that's closer to one sample without all the models.
- im3w1l 6y agoI think he is saying that a competition where both sides are allowed to bring a lot of preconceptions into the learning task is an interesting case where people currently beat the machines by a lot.
- fallingfrog 6y agoIt’s not a fair comparison, of course, but you know, I don’t think the public cares as much about whether it’s fair as they care about how well it works. I guess what I’m saying is that to match human performance you’d need all that prior understanding that you just listed. Maybe that’s what’s missing from our models.
- Grimm1 6y agoWhen this was discussed about two months ago the conclusion I took away was not very many at the moment beyond somewhat formal equivalence. https://news.ycombinator.com/item?id=25314830 https://news.ycombinator.com/item?id=25314830
- 6gvONxR4sf7o 6y agoNeither. It's more akin to the fundamental theorem of calculus. If you follow your data from point A to point B along some differential path, you can sum up/integrate the steps to follow your data in a single jump from point A to point B directly (with the jump in terms of those integrals). It's a super interesting viewpoint on gradient descent and models that use it that could be really useful in looking at and understanding those models abstractly, but isn't saying anything about suitability for different tasks.
- 6gvONxR4sf7o 6y agoDiscussion of the actual paper here: https://news.ycombinator.com/item?id=25314830 https://news.ycombinator.com/item?id=25314830 It's a really neat one for people who care about what's going on under the hood, but not immediately applicable to the more applied folks. I saw some good quotes at the time to the tune of "I can't wait to see the papers citing this one in a year or two."
- Fordec 6y agoI'm not in this subsection of the industry, so take this with a heaping grain of salt. I'm just using my own biases here. But I get the sense that this is one of a number of big pieces of the puzzle of how current cutting edge models can be cut down in operational size. I have a feeling that new methods of creating models will stick around, but any deep learning model can be reduced to a less resource intensive approximation through findings such as this. And if (IF!) so a lot of AI hype is going to hit a real post-hype cycle plateau of productivity because all this stuff will become easier and cost effective to deploy to the real world.
- scythmic_waves 6y agoThe paper discussed showed on reddit a few months back [1]. Another paper showed up shortly after claiming the exact opposite [2]. Some discussion of contradiction this can be found in this part of the thread: [3]. I myself am very interested in [2]. It's fairly dense, but I've been meaning to go through it and the larger Tensor Programs framework ever since. [1] https://www.reddit.com/r/MachineLearning/comments/k7wj5s/r_every_model_learned_by_gradient_descent_is/ https://www.reddit.com/r/MachineLearning/comments/k7wj5s/r_e... [2] https://www.reddit.com/r/MachineLearning/comments/k8h01q/r_wide_neural_networks_are_feature_learners_not/ https://www.reddit.com/r/MachineLearning/comments/k8h01q/r_w... [3] https://www.reddit.com/r/MachineLearning/comments/k8h01q/r_wide_neural_networks_are_feature_learners_not/geya33m/ https://www.reddit.com/r/MachineLearning/comments/k8h01q/r_w...
- Grimm1 6y agoThank you! Number 2 was exactly what I was looking for and I just couldn't find the link.
- ur-whale 6y ago> Gradient Descent Models Are Kernel Machines ... that also happen to actually work.
- tlb 6y agoIf you're surprised by this result because you've used kernel machines and didn't find them very good at generalization, keep in mind that this assumes a kernel function that accurately reflects similarity of input samples. Most work with kernel machines just uses Euclidean distance. For instance, in an image recognition model it would have to identify 2 images with dogs as more similar than an image with a dog and an image with a cat. With a sufficiently magical kernel function, indeed you can get great results with a kernel machine. But it's not so easy to write a kernel function for a domain like image processing, where shifts, scales, and small rotations shouldn't affect similarity much. Let alone for text processing, where it should recognize 2 sentences with similar meaning as similar.
- yudlejoza 6y agoI may or may not be surprised by the result, I'm definitely not surprised by the 'Thing A is thing B' in machine learning. Every tom, machine-learner, and harry, is an expert on proving to the whole world that thing-A is thing-B. The only problem is people don't hire them with a million dollars a year total compensation.
- mywittyname 6y ago>With a sufficiently magical kernel function, indeed you can get great results with a kernel machine. But it's not so easy to write a kernel function for a domain like image processing, where shifts, scales, and small rotations shouldn't affect similarity much. Let alone for text processing, where it should recognize 2 sentences with similar meaning as similar. I think the key issue at hand is that gradient descent is easier to train than a model using kernel functions. Someone could absolutely devise a mechanism for back propagation of errors with kernel functions, but at that point, it is basically a neural network.
- SubiculumCode 6y agosomedays I feel that, in the final analysis, everything is just linear regression.
- tqi 6y agohttps://twitter.com/theotheredmund/status/1349453230762196992 https://twitter.com/theotheredmund/status/134945323076219699...
- wenc 6y agoAs someone who has studied nonlinear nonconvex optimization, I don't think linear regression is the final word here. In the universe of optimization problems for curve-fitting, the linear case is only one case (albeit a very useful one). In many cases though, it is often insufficient. The next step is piecewise linearity and then convexity. It is said that convexity is a way to weaken a linearity requirement while leaving a problem tractable. Many real world systems are nonlinear (think physics models), and often nonconvex. You can approximate them using locally linear functions to be sure, but you lose a lot of fidelity in the process. Sometimes this is ok, sometimes this is not, so it depends on the final application. It happens that linear regression is good enough for a lot of stuff out there, but there are many places where it doesn't work.
- nightcracker 6y ago> Many real world systems are nonlinear (think physics models) Technically, only if you don't zoom in too far, quantum mechanics is linear.
- ks1723 6y agoThis is a little too simplistic/misleading - yes, QM is linear in terms of states, vector spaces and operators acting on states (its just functional analysis [1]) - it’s not linear in terms of how probability distributions and states behave under the action of forces and multi-body interactions (see, eg correlated phenomena like phase transitions, magnetism, superconductivity etc) [1] https://en.m.wikipedia.org/wiki/Functional_analysis https://en.m.wikipedia.org/wiki/Functional_analysis
- robrenaud 6y agoYannic Kilcher's paper explained series covers this pretty paper well. I feel like I have a decent understanding of it just after watching the video with a few pauses/rewinds. https://www.youtube.com/watch?v=ahRPdiCop3E https://www.youtube.com/watch?v=ahRPdiCop3E
- kdisorte 6y agoQuoting an expert from the last time this was posted: So at the end, it rephrased a statement from "Neural Tangent Kernel: Convergence and Generalization in Neural Networks" [https://arxiv.org/abs/1806.07572 https://arxiv.org/abs/1806.07572], besides in a way which is kind of miss-leading. The assertion is known by the community at least since 2018, if not even well before. I find this article and the buzz around a little awkward.
- gbrown 6y agoIs this now considered correct/acceptable terminology? Back in my day optimization techniques weren’t the same thing as models. You can get point estimates for many completely different models using gradient descent. The article clarifies what is meant in the first line, but it still seems like a very poor choice of words. Now get off my lawn.
- deleted 6y ago[deleted]
- mysterEFrank 6y agoI think it's clear that there's a strong relationship between all of these methods. Papers proving that Neural Networks are GPs came out a year or two ago, for example. Deep learning is more useful in the end because kernel machines and gaussian processes (classically) store all samples in a dataset and require expensive pairwise comparison. Deep learning/gradient descent doesn't.
- rq1 6y agoI don’t want to seem like minimising (if I may say) all these works and similar ones. This is not surprising at all and they’re not new results actually. Cf. Donsker-like theorems.
- shenberg 6y agoThere's some super-interesting recent empirical work from MS research that makes this perspective less likely to be a useful one. In short, the behavior of neural networks in a few regimes makes this interpretation of them less likely, specifically: ensembling the same network trained multiple times separately improves results compared to training multiple networks concurrently, knowledge distillation works, and self-distillation works. The blog post[1] has a good intuitive explanation of the matter and a link to the paper. [1] https://www.microsoft.com/en-us/research/blog/three-mysteries-in-deep-learning-ensemble-knowledge-distillation-and-self-distillation/ https://www.microsoft.com/en-us/research/blog/three-mysterie...
- orange3xchicken 6y agoTo add on: Zeyuan Allen-Zhu and Yuanzhi Li had a great theoretical paper earlier last year proving a fundamental discrepancy between the concept classes learnable by kernel methods and deep learning models. https://arxiv.org/abs/2001.04413 https://arxiv.org/abs/2001.04413
- thwoeir324234 6y agoThe form derived in Pedro's paper is not a standard kernel machine - the coefficients depend on the example, and even the form of the Kernel machine is obtained rather unnaturally by pushing one kernel term into the coefficient and keeping the other outside of it.
- laingc 6y ago* for a very specific, nonstandard definition of a kernel machine...