8 ms·
Every model learned by gradient descent is approximately a kernel machine (2020)
- nickpsecurity 3y agoThanks for the submission. I keep an eye out for connections to potentially cheaper and simpler methods. The new comparison reminds me of this old paper which compared them to Gaussian networks: https://arxiv.org/abs/1711.00165 https://arxiv.org/abs/1711.00165 For causality, I'm also keeping an eye out for ways to tie DNN research back into decision trees, esp probabilistic and nested. Or fuzzy logic. Here's an example I saw moving probabilistic methods into decision trees: http://www.gatsby.ucl.ac.uk/~balaji/balaji-phd-thesis.pdf http://www.gatsby.ucl.ac.uk/~balaji/balaji-phd-thesis.pdf Many of these sub-fields develop separately in their own ways. Gotta wonder what recent innovations in one could be ported to another.
- yazzku 3y agoThis seems like the key research to me if we want any shot at preventing the technology from being locked away behind big corp API walls interfacing giant data centers. Anything that removes the bloat and mysticism from the models so they can be scaled down and run on the little guy's computer is orders of magnitude more progress in my opinion than, e.g., increasing the token window by some epsilon.
- westurner 3y agoDoes a given model converge after Gaussian blurring? What does it do in the presence of noise, given the curse of dimensionality? OpenCog integrates PLN and MOSES (~2005). "Interpretable Model-Based Hierarchical RL Using Inductive Logic Programming" (2021) https://news.ycombinator.com/item?id=37463686 https://news.ycombinator.com/item?id=37463686 : > https://en.wikipedia.org/wiki/Probabilistic_logic_network https://en.wikipedia.org/wiki/Probabilistic_logic_network : >> The basic goal of PLN is to provide reasonably accurate probabilistic inference in a way that is compatible with both term logic and predicate logic, and scales up to operate in real time on large dynamic knowledge bases Asmoses updates an in-RAM (*) online hypergraph with the graph relations it learns. CuPy wraps CuDNN. Re: quantum logic and quantum causal inference: https://news.ycombinator.com/item?id=38721246 https://news.ycombinator.com/item?id=38721246 From https://news.ycombinator.com/item?id=39255303 https://news.ycombinator.com/item?id=39255303 : > Is Quantum Logic the correct propositional logic? Is Quantum Logic a sufficient logic for all things? A quantum ML task: Find all local and nonlocal state linkages within the presented observations And then also do universal function approximation
- westurner 3y agoBut also biological neurological systems; Coping strategies: https://en.wikipedia.org/wiki/Coping https://en.wikipedia.org/wiki/Coping Defense Mechanisms > Vaillant's categorization > Level 4: mature: https://en.wikipedia.org/wiki/Defence_mechanism#Level_4:_mature https://en.wikipedia.org/wiki/Defence_mechanism#Level_4:_mat... "From Comfort Zone to Performance Management" suggests that the Carnall coping cycle coincides with the TPR curve (Transforming, Performing, Reforming, adjourning); that coping with change in systems is linked with performance. And Consensus; social with nonlinear feedback and technological. What are the systems thinking advantages in such fields of study? Systems theory > See also > Glossary,: https://en.wikipedia.org/wiki/Systems_theory#See_also https://en.wikipedia.org/wiki/Systems_theory#See_also
- westurner 3y agoWrong thread, my mistake; this comment was for this thread: https://news.ycombinator.com/context?id=39504104 https://news.ycombinator.com/context?id=39504104
- Legend2440 3y agoThis is neither cheaper or simpler. Kernel methods are much more computationally expensive because the cost scales with the square of the dataset size. Neural networks are actually decently efficient, they mostly seem slow because we apply them to problems (like modeling the entire internet) that are just huge.
- zaptrem 3y agoAfaik this was stated in my Intro to ML course. A kernel machine can do anything when the similarity function has infinite dimensions. Similarly, I think they mentioned an infinitely wide MLP is also all you need. Also, this all breaks down when you introduce reinforcement learning methods.
- dartos 3y agoWhat do you mean breaks down?
- ultra_nick 3y agoBreaks. Methods that expect the same input to map to the same output don't work with feedback.
- zaptrem 3y agoThere's no longer a concept of training examples to be close to, since it's just going along the gradient of high reward actions in the RL environment and going away from those with low reward.
- nsenifty 3y ago> Afaik this was stated in my Intro to ML course. Isbell?
- mxwsn 3y agoThis connects to a suspicion I have that a lot of the most impressive results, that have caught public attention, for LLMs and Sora are due to memorization. This is not to say that they can't generalize or mix learned patterns - I think they can do that too, which is quite impressive to a researcher, but these are demonstrated on tasks that the public might not care as much about, and the performance there isn't always great as well. Combined with opacity in train/test splits, this suggests a type of data laundering, where the extreme is that much of Sora (for example) is regurgitating things seen in training, which was actually made perhaps by a human using a game engine, or drone video. But it's only news because a model generated it. Of course, I don't have strong evidence for this. But one of the most impressive parts of Sora is the level of detail, much of which is not specified by the text prompt, and still can't fully be accounted for by our knowledge that OAI expands text prompts for image/ video generation behind the scenes. Where is this precise detail coming from exactly? I suggest it's from memorization.
- quanto 3y agoThen, by your standards, where do most human occupations lie? Plumbers, lawyers, and engineers? Aren't they mostly regurgitating (by your standards) what they have seen or learned before?
- vasilipupkin 3y agobingo, I think it's a weird gotcha. Most human occupations require years of training exactly for this purpose, so that humans can just regurgitate data they have seen in training. Very few humans ever produce truly novel insights. you wouldn't hire a divorce lawyer for a corporate merger and vice versa
- dartos 3y agoI think that’s a gross oversimplification. Have you been a plumber or an electrician? Different buildings require different, occasionally novel, solutions. Not all are textbook (most aren’t.) Not all innovations are published or even widely communicated. Just like in programming, the devil is in the details, otherwise we’d all be using ruby in rails for our web servers, since regurgitating how to build a CRUD app is all you’d need.
- cubefox 3y ago(2020)
- quanto 3y agoThis paper was submitted in 2020, and the fact that NN, in the most simple case, is a kernal-learning algo was known even before that. For those who are more familiar with this subtopic, what contribution does this paper make?
- nextaccountic 3y agoDiscussion at the time https://news.ycombinator.com/item?id=25314830 https://news.ycombinator.com/item?id=25314830
- dang 3y agoThanks! Macroexpanded: Every Model Learned by Gradient Descent Is Approximately a Kernel Machine - https://news.ycombinator.com/item?id=25314830 https://news.ycombinator.com/item?id=25314830 - Dec 2020 (107 comments)
- tho2iu34234 3y ago"Remark-1" says it actually is not. Surprised Pedro is pushing this as a Kernel Machine (the need for constant coefficients is a key requirement).
- Straw 3y agoThis keeps popping up but while technically true, its essentially nonsense- normally when people talk about kernel machines, the kernel doesn't depend on the data, or at least not very much- we might use a Gaussian kernel and tune the radius or even the covariance. This construction has a kernel which depends on the entire training trajectory of the neural network! So its completely unclear what's happening, all of the interesting parts may have just moved into the kernel. So basically this tells us nothing- we can't just add a new data point as in a kernel method, incorporating it just by adding its interaction- every new data point changes the whole training trajectory so could completely change the resulting kernel.
- mildmay_rider 3y agoYou may enjoy some works on the connections between gaussian processes, neural networks, and linear(in rkhs) models. https://papers.nips.cc/paper_files/paper/2019/hash/39d929972619274cc9066307f707d002-Abstract.html https://papers.nips.cc/paper_files/paper/2019/hash/39d929972... https://papers.nips.cc/paper_files/paper/1996/hash/ae5e3ce40e0404a45ecacaaf05e5f735-Abstract.html https://papers.nips.cc/paper_files/paper/1996/hash/ae5e3ce40... https://arxiv.org/abs/1711.00165 https://arxiv.org/abs/1711.00165
- Straw 3y agoI'm familiar with the NN-as-GP papers. In practice we don't train infinite width neural networks!
- Craig_Rodriguez 3y ago[dead]
- latenightcoding 3y agoI've seen this shared here and there, but it's not original research, Pedro Domingos doesn't do much of that anymore, mostly spends his time trolling on twitter an calling everybody "woke"
- esafak 3y agoI did not realize he was retired.
- rich_sasha 3y agoI suppose every single ML method, ever, is effectively memorisation. The crux is the feature representation is builds to aid the memorisation. So take OLS, or a decision tree / forest. When you evaluate them, they look at some feature space, then compare to some fitted parameters (i.e. memorises result) and produce an output. Methods such as SVM or Lasso are also memorisation, except with a clever feature mapping. Instead of memorising outputs for every sample, the memories them by some learned feature sunset or transformation. Perhaps the same thing happens for LLMs or NNs (I'm still reading the paper) but if so, this is just the name of the game in ML, it seems. Arguably that's what we humans do too. We are capable of creative thinking, whatever that means, but 90% of our thought process seems to be cache recall, and intelligence seems to correspond well to having a large and diverse cache. Many disciplines that require creative thinking to understand, like maths or playing instruments, seem to improve on repetitive practice that makes us memorize patterns.
- blackbear_ 3y agoYou seem to be implying that is a negative thing, but it's pretty hard to imagine a form of intelligence that does not require any form of memory at all. In case of AI, at the very least you need to memorize the executable machine code, and its analogous in living organisms is whatever nervous system was initialized at birth.
- rich_sasha 3y agoI don't mean it as a negative, just not unique to gradient methods.
- Benjaminsen 3y agoEven if that's the case, a sufficiently large hash table is indistinguishable from an intelligence by a lower intelligence.
- iamgopal 3y agoWe have seen that some human are capable of doing complex ( square root ) maths in their head, can NN be trained to do this accurately ?
- math_dandy 3y agoPerhaps memory plays a much larger role in human intelligence, reasoning, and cognition than we give it credit for.
- cma 3y agoDoesn't the later work on tensor programs show this is wrong? It was show there is a region of hyper parameters that will be kernel methods, and another region where it will learn more meaningful features.
- nobodyandproud 3y agoThere seems to be a not-so-friendly academic or research rivalry between the SVM crowd and the ANN crowd. Am I imagining things?
- dr_kiszonka 3y agoI would say a lot of stats and ML folks strongly dislike anything related to deep learning.