6 ms·
What do you think about the papers showing mathematical proofs that GNNs (i.e. GATs/transformers) are dynamic programmers and therefore perform algorithmic reas
by chaxor 3y ago
What do you think about the papers showing mathematical proofs that GNNs (i.e. GATs/transformers) are dynamic programmers and therefore perform algorithmic reasoning?
The fact that these systems can extrapolate well beyond their training data by learning algorithms is quite different than what has come before, and anyone stating that they "simply" predict next token is severely shortsighted. Things don't have to be 'brain-like' to be useful, or to have capabilities of reasoning, but we have evidence that these systems have aligned well with reasoning tasks, perform well at causal reasoning, and we also have mathematical proofs that show how.
So I don't understand your sentiment.
- agentofoblivion 3y agoGive me a break. Very interesting theoretical work and all, but show me where it's actually being used to do anything of value, beyond publication fodder. You could also say MLPs are proved to be universal approximators, and can therefore model any function, including the one that maps sensory inputs to cognition. But the disconnect between this theory and reality is so great that it's a moot point. No one uses MLPs this way for a reason. No one uses GATs in systems that people are discussing right now either. GATs rarely even beat GCNs by any significant margin in graph benchmarks.
- chaxor 3y agoAre you saying that the new mathematical theorems that were proven using GNNs from Deepmind were not useful? There were two very noteworthy (Perhaps Nobel prize level?) breakthroughs in two completely different fields of mathematics (knot theory and representation theory) by using these systems. I would certainly not call that "useless", even if they're not quite Nobel-prize-worthy. Also, "No one uses GATs in systems people discuss right now" ... Transformerare GATs (with PE) ... So, you're incredibly wrong.
- agentofoblivion 3y agoYou’re drinking from the academic marketing koolaid. Please tell me: where are these methods being applied in AI systems today? And I’m so tired of this “transformers are just GNNs” nonsense that Petar has been pushing (who happens to have invented GATs and has a vested interest in overstating their importance). Transformers are GNNs in only the most trivial way: if you make the graph fully connected and allow everything to interact with everything else. I.e., not really a graph problem. Not to mention that the use of positional encodings breaks the very symmetry that GNNs were designed to preserve. In practice, no one is using GNN tooling to build transformers. You don’t see PyTorch geometric or DGL in any of the code bases. In fact, you see the opposite: people exploring transformers to replace GNNs in graph problems and getting SOTA results. It reminds me of people that are into Bayesian methods always swooping in after some method has success and saying, “yes, but this is just a special case of a Bayesian method we’ve been talking about all along!” Yes, sure, but GATs have had 6 years to move the needle, and they’re no where to be found within modern AI systems that this thread is about.
- uh_uh 3y agoI just don't get how the average HN commenter thinks (and gets upvoted) that they know better than e.g. Ilya Sutskever who actually, you know, built the system. I keep reading this "it just predicts words, duh" rhetoric on HN which is not at all believed by people like Ilya or Hinton. Could it be that HN commenters know better than these people?
- dmreedy 3y agoI am reminded of the Mitchell and Webb "Evil Vicars" sketch. "So, you've thought about eternity for an afternoon, and think you've come to some interesting conclusions?"
- RandomLensman 3y agoThat is the wrong discussion. What are their regulatory, social, or economic policy credentials?
- uh_uh 3y agoI'm not suggesting that they have any. I was reacting to srslack above making _technical_ claims why LLMs can't be "generalized and adaptable intelligence" which is not shared by said technical experts.
- shafyy 3y agoThe thing is, experts like Ilya Sutskever are so deep in that shit that they are heavily biased (from a tech and social/economic) perspective. Furthermore, many experts are wrong all the time. I don't think the average HN commenter claims to be better at building these system than an expert. But to criticize, especially critic on economic, social, and political levels, one doesn't need to be an expert on LLMs. And finally, what the motivation of people like Sam Altman and Elon Musk is should be clear to everbody with a half a brain by now.
- uh_uh 3y agosrslack above was making technical claims why LLMs can't be "generalized and adaptable intelligence". To make such statements, it surely helps if you are a technical expert at building LLMs.
- rdedev 3y agoTo be fair LLMs are predicting the next token. It's just that to get better and better predictions it needs to understand some level of reasoning and math. However it feels to me that a lot of this reasoning is brute forced from the training data. Like chatgpt gets some things wrong when adding two very large numbers. If it really knew the algorithm for adding two numbers it shouldn't be making them in the first place. I guess same goes for issues like hallucinations. We can keep pushing the envelope using this technique but I'm sure we will hit a limit somewhere
- chaxor 3y agoOf course it predict the next token. Every single person on earth knows that so it's not worth repeating at all. As for the fact that it gets things wrong sometimes - sure, this doesn't say it actually learned every algorithm (in whichever model you may be thinking about). But the nice thing is that we now have this proof via category theory, and it allows us to both frame and understand what has occurred, and to consider how to align the systems to learn algorithms better.
- rdedev 3y agoThe fact that it sometimes fails simple algorithms for large numbers but shows good performance in other complex algorithms with simple inputs seems to me that something on a fundamental level is still insufficient
- zamnos 3y agoInsufficient for what? Humans regularly fail simple algorithms for small numbers, nevermind large numbers and complex algorithms
- starlust2 3y agoYou're focusing too much on what the LLM can handle internally. No LLMs aren't good at math, but they understand mathematic concepts and can use a program or tool to perform calculations. Your argument is the equivalent of saying humans can't do math because they rely on calculators. In the end what matters is whether the problem is solved, not how it is solved. (assuming that the how has reasonable costs)
- pdonis 3y ago> What do you think about the papers showing mathematical proofs that GNNs (i.e. GATs/transformers) are dynamic programmers and therefore perform algorithmic reasoning? Do you have a reference?
- felipemnoa 3y ago>>What do you think about the papers showing mathematical proofs that GNNs (i.e. GATs/transformers) are dynamic programmers and therefore perform algorithmic reasoning? Do you mind linking to one of those papers?
- joaogui1 3y agoThe paper shows the equivalence for specific networks, it doesn't say every GNN (and as such transformers) are Dynamic Programmers. Also the models are explicitly trained on that task, in a regime quite different from ChatGPT. What the paper shows and the possibility of LLMs being able to reason are pretty much completely independent from each other