5 ms·

LLMs are incapable of solving even simple logic puzzles or maths puzzles they haven't seen before, they don't have a model of the world which is key to intellig

LLMs are incapable of solving even simple logic puzzles or maths puzzles they haven't seen before, they don't have a model of the world which is key to intelligence. What they are good at is reproducing things in their dataset with slight modifications and (sometimes) responding to queries well which make them seem creative but there is no understanding or intelligence there, in spite of appearances. They are very good at fooling people; perhaps Turing's Test is not a good measure of intelligence after all, it can easily be gamed and we find it hard to differentiate apparent facility with language and intelligence/knowledge.

jychang 2y ago
LLMs clearly do have a world model though. They represent those ideas at higher level features in the feedforward layer. The lower level layers are neurons that describe words, syntax, and local structures in the text, while the upper levels capture more abstract ideas, such as semantic meaning, relationships between concepts, and even implicit reasoning patterns.
- troupo 2y ago
  Funny how literally nothing of what you wrote is happening.
  - gmadsen 2y ago
    https://www.anthropic.com/news/mapping-mind-language-model https://www.anthropic.com/news/mapping-mind-language-model It is clearly happening as shown by numerous papers studying it. Here is a popular one by anthropic
    - troupo 2y ago
      > https://www.anthropic.com/news/mapping-mind-language-model https://www.anthropic.com/news/mapping-mind-language-model I wouldn't read into marketing materials by the people whose funding depends on hype. Nothing in the link you provided is even close to "neurons, model of the world, thinking" etc. It literally is "in our training data similar concepts were clustered with some other similar concepts, and manipulating these clusters lead to different outcomes".
      - dleary 2y ago
        > It literally is "in our training data similar concepts were clustered with some other similar concepts, and manipulating these clusters lead to different outcomes". Recognizing concepts, grouping and manipulating similar concepts together, is what “abstraction” is. It's the fundamental essence of both "building a world model" and "thinking". > Nothing in the link you provided is even close to "neurons, model of the world, thinking" etc. I really have no idea how to address your argument. It’s like you’re saying, “Nothing you have provided is even close to a model of the world or thinking. Instead, the LLM is merely building a very basic model of the world and performing very basic reasoning”.
        troupo 2y ago
        > Recognizing concepts, grouping and manipulating similar concepts together, Once again, it does none of those things. The training dataset has those concepts grouped together. The model recognizes nothing, and groups nothing > I really have no idea how to address your argument. It’s like you’re saying, No. I'm literally saying: there's literally nothing to support your belief that there's anything resembling understanding of the world, having a world model, neurons, thinking, or reasoning in LLMs.
        dleary 2y ago
        > there's literally nothing to support your belief that there's anything resembling understanding of the world, having a world model, neurons, thinking, or reasoning in LLMs. The link mentions "a feature that triggers on the Golden Gate Bridge". As a test case, I just drew this terrible doodle of the Golden Gate Bridge in MS paint: https://imgur.com/a/1TJ68JU https://imgur.com/a/1TJ68JU I saved the file as "a.png", opened the chatgpt website, started a new chat, uploaded the file, and entered, "what is this?" It had a couple of paragraphs saying it looked like a suspension bridge. I said "which bridge". It had some more saying it was probably the GGB, based on two particular pieces of evidence, which it explained. https://imgur.com/a/sWwWlxO https://imgur.com/a/sWwWlxO > The model recognizes nothing, and groups nothing Then how do you explain the interaction I had with chatgpt just now? It sure looks to me like it recognized the GGB from my doodle.
    - dartos 2y ago
      That’s a marketing article, bub. You should learn a bit about media literacy.
      - gmadsen 2y ago
        the paper is clearly linked in the marketing article, bub
        dartos 2y ago
        A (afaics non peer-reviewed) paper published by a random organization endorsed by Anthropic does not proof make. In fact, it still very much seems like marketing. Especially since the paper was made in association with Anthropic. Again. Learn some media literacy.
Applejinx 2y ago
Along these lines one model that might help is to consider LLMs 'wikipedia of all possible correct articles'. Start with Wikipedia and assume (already a tricky proposition!) that it's perfectly correct. Then, begin resynthesizing articles based on what's already there. Do your made-up articles have correctness? I'm going to guess that sometimes they will: driven onto areas where there's no existing article, some of the time you'll get made-up stuff that follows the existing shapes of correct articles and produces articles that upon investigation will turn out to be correct. You'll also reproduce existing articles: in the world of creating art, you're just ripping them off, but in the world of Wikipedia articles you're repeating a correct thing (or the closest facsimile that process can produce) When you get into articles on exceptions or new discoveries, there's trouble. It can't resynthesize the new thing: the 'tokens' aren't there to represent it. The reality is the hallucination, but an unreachable one. So the LLMs can be great at fooling people by presenting 'new' responses that fall into recognized patterns because they're a machine for doing that, and Turing's Test is good at tracking how that goes, but people have a tendency to think if they're reading preprogrammed words based on a simple algorithm (think 'Eliza') they're confronting an intelligence, a person. They're going to be historically bad at spotting Holmes-like clues that their expected 'pattern' is awry. The circumstantial evidence of a trout in the milk might lead a human to conclude the milk is adulterated with water as a nefarious scheme, but to an LLM that's a hallucination on par with a stone in the milk: it's going to have a hell of a time 'jumping' to a consistent but very uncommon interpretation, and if it does get there it'll constantly be gaslighting itself and offering other explanations than the truth.
rcxdude 2y ago
I think it's not very helpful to just declare that such a model doesn't exist: there's a decent amount of evidence that LLMs do in fact form models of the world internally, and use them during inference. However, while these models are very large and complex, they aren't necessarily accurate and LLMs struggle with actually manipulating them at inference time, forming new models or adjusting existing ones is generally something they are quite bad at at those stages (which generally results in the 'high knowledge' which impresses people and is often confused with intelligence, while they're still fundamentally quite dumb despite having a huge depth of knowledge: I don't think it's something you can categorically say 'zero intelligence' - even relatively simpler and less capable systems can be said to have some intelligence, it's just in many aspects LLM intelligence is still worse than a good fraction of mammals)
- grey-area 2y ago
  What evidence are you referring to? I've seen AI firms desperate for relevance and the next advance implying that thinking is going on and talking about it a lot in those terms, but no actual evidence of it. I wouldn't say zero intelligence, but I wouldn't describe such systems as intelligent, I think it misrepresents them, they do as you say have a good depth of knowledge and are spectacular at reproducing a simulacrum of human interactions and creations, but they have been a lesson for many of us that token manipulation is not where intelligence resides.
fragmede 2y ago
> they don't have a model of the world Must it have one? The words "artificial intelligence" are a poor description of a thing when we've not rigorously defined it. It's certainly artificial, there's no question about that, but is it intelligent? It can do all sorts of things that we consider a feature of intelligence and pass all sorts of tests, but it also falls down flat on its face when prompted with a just-so brainteaser. It's certainly useful, for some people. If, by having inhaled all of the Internet and written books that have been scanned as its training data, it's able to generate essays on anything and everything, at the drop of a hat, why does it matter if we can find a brainteaser it hasn't seen yet? It's like it has a ginormous box of Legos, and it can build whatever your ask for with these Lego blocks, but pointing out it's unable create its own Lego blocks from scratch has somehow become critically important to point out, as if that makes this all total dead end and it's all a waste of money omg people wake up oh if only they'd listen to me. Why don't people listen to me? Crows are believed to have a theory of mind, and they can count up to 30. I haven't tried it with Claude, but I'm pretty sure it can count at least that high. LLMs are artificial, they're alien, of course they're going to look different. In the analogy where they're simply a next word guesser, one imagines standing at a fridge with a bag of magnetic words, and just pulling a random one from the bag to make ChatGPT. But when you put your hand inside a bag inside a bag inside a bag, twenty times (to represent the dozens of layers in an LLM model), and there are a few hundred million pieces in each bag (for parameters per layer), one imagines that there's a difference; some sort of leap, similar to when life evolved from being a single celled bacterium to a multi-cellular organism. Or maybe we're all just rubes, and some PhD's have conned the world into giving them a bunch of money, because they figured out how to represent essays as a math problem, then wrote some code to solve them, like they did with chess.
- dartos 2y ago
  There’s a bit of truth to all of what you said. These tools aren’t useless, obviously. But people do really learn hard into confirmation bias and/or personification when it comes to LLMs. I believe it’s entirely because of the term “artificial intelligence” that there is such a divide. If we called them “large statistical language models” instead, nobody would be having this discussion.
- 2y ago

jychang 2y ago

LLMs clearly do have a world model though. They represent those ideas at higher level features in the feedforward layer. The lower level layers are neurons that describe words, syntax, and local structures in the text, while the upper levels capture more abstract ideas, such as semantic meaning, relationships between concepts, and even implicit reasoning patterns.

troupo 2y ago
Funny how literally nothing of what you wrote is happening.
- gmadsen 2y ago
  https://www.anthropic.com/news/mapping-mind-language-model https://www.anthropic.com/news/mapping-mind-language-model It is clearly happening as shown by numerous papers studying it. Here is a popular one by anthropic
  - troupo 2y ago
    > https://www.anthropic.com/news/mapping-mind-language-model https://www.anthropic.com/news/mapping-mind-language-model I wouldn't read into marketing materials by the people whose funding depends on hype. Nothing in the link you provided is even close to "neurons, model of the world, thinking" etc. It literally is "in our training data similar concepts were clustered with some other similar concepts, and manipulating these clusters lead to different outcomes".
    - dleary 2y ago
      > It literally is "in our training data similar concepts were clustered with some other similar concepts, and manipulating these clusters lead to different outcomes". Recognizing concepts, grouping and manipulating similar concepts together, is what “abstraction” is. It's the fundamental essence of both "building a world model" and "thinking". > Nothing in the link you provided is even close to "neurons, model of the world, thinking" etc. I really have no idea how to address your argument. It’s like you’re saying, “Nothing you have provided is even close to a model of the world or thinking. Instead, the LLM is merely building a very basic model of the world and performing very basic reasoning”.
      - troupo 2y ago
        > Recognizing concepts, grouping and manipulating similar concepts together, Once again, it does none of those things. The training dataset has those concepts grouped together. The model recognizes nothing, and groups nothing > I really have no idea how to address your argument. It’s like you’re saying, No. I'm literally saying: there's literally nothing to support your belief that there's anything resembling understanding of the world, having a world model, neurons, thinking, or reasoning in LLMs.
        dleary 2y ago
        > there's literally nothing to support your belief that there's anything resembling understanding of the world, having a world model, neurons, thinking, or reasoning in LLMs. The link mentions "a feature that triggers on the Golden Gate Bridge". As a test case, I just drew this terrible doodle of the Golden Gate Bridge in MS paint: https://imgur.com/a/1TJ68JU https://imgur.com/a/1TJ68JU I saved the file as "a.png", opened the chatgpt website, started a new chat, uploaded the file, and entered, "what is this?" It had a couple of paragraphs saying it looked like a suspension bridge. I said "which bridge". It had some more saying it was probably the GGB, based on two particular pieces of evidence, which it explained. https://imgur.com/a/sWwWlxO https://imgur.com/a/sWwWlxO > The model recognizes nothing, and groups nothing Then how do you explain the interaction I had with chatgpt just now? It sure looks to me like it recognized the GGB from my doodle.
  - dartos 2y ago
    That’s a marketing article, bub. You should learn a bit about media literacy.
    - gmadsen 2y ago
      the paper is clearly linked in the marketing article, bub
      - dartos 2y ago
        A (afaics non peer-reviewed) paper published by a random organization endorsed by Anthropic does not proof make. In fact, it still very much seems like marketing. Especially since the paper was made in association with Anthropic. Again. Learn some media literacy.

Applejinx 2y ago

Along these lines one model that might help is to consider LLMs 'wikipedia of all possible correct articles'. Start with Wikipedia and assume (already a tricky proposition!) that it's perfectly correct. Then, begin resynthesizing articles based on what's already there. Do your made-up articles have correctness? I'm going to guess that sometimes they will: driven onto areas where there's no existing article, some of the time you'll get made-up stuff that follows the existing shapes of correct articles and produces articles that upon investigation will turn out to be correct. You'll also reproduce existing articles: in the world of creating art, you're just ripping them off, but in the world of Wikipedia articles you're repeating a correct thing (or the closest facsimile that process can produce) When you get into articles on exceptions or new discoveries, there's trouble. It can't resynthesize the new thing: the 'tokens' aren't there to represent it. The reality is the hallucination, but an unreachable one. So the LLMs can be great at fooling people by presenting 'new' responses that fall into recognized patterns because they're a machine for doing that, and Turing's Test is good at tracking how that goes, but people have a tendency to think if they're reading preprogrammed words based on a simple algorithm (think 'Eliza') they're confronting an intelligence, a person. They're going to be historically bad at spotting Holmes-like clues that their expected 'pattern' is awry. The circumstantial evidence of a trout in the milk might lead a human to conclude the milk is adulterated with water as a nefarious scheme, but to an LLM that's a hallucination on par with a stone in the milk: it's going to have a hell of a time 'jumping' to a consistent but very uncommon interpretation, and if it does get there it'll constantly be gaslighting itself and offering other explanations than the truth.

rcxdude 2y ago

I think it's not very helpful to just declare that such a model doesn't exist: there's a decent amount of evidence that LLMs do in fact form models of the world internally, and use them during inference. However, while these models are very large and complex, they aren't necessarily accurate and LLMs struggle with actually manipulating them at inference time, forming new models or adjusting existing ones is generally something they are quite bad at at those stages (which generally results in the 'high knowledge' which impresses people and is often confused with intelligence, while they're still fundamentally quite dumb despite having a huge depth of knowledge: I don't think it's something you can categorically say 'zero intelligence' - even relatively simpler and less capable systems can be said to have some intelligence, it's just in many aspects LLM intelligence is still worse than a good fraction of mammals)

grey-area 2y ago
What evidence are you referring to? I've seen AI firms desperate for relevance and the next advance implying that thinking is going on and talking about it a lot in those terms, but no actual evidence of it. I wouldn't say zero intelligence, but I wouldn't describe such systems as intelligent, I think it misrepresents them, they do as you say have a good depth of knowledge and are spectacular at reproducing a simulacrum of human interactions and creations, but they have been a lesson for many of us that token manipulation is not where intelligence resides.

fragmede 2y ago

> they don't have a model of the world Must it have one? The words "artificial intelligence" are a poor description of a thing when we've not rigorously defined it. It's certainly artificial, there's no question about that, but is it intelligent? It can do all sorts of things that we consider a feature of intelligence and pass all sorts of tests, but it also falls down flat on its face when prompted with a just-so brainteaser. It's certainly useful, for some people. If, by having inhaled all of the Internet and written books that have been scanned as its training data, it's able to generate essays on anything and everything, at the drop of a hat, why does it matter if we can find a brainteaser it hasn't seen yet? It's like it has a ginormous box of Legos, and it can build whatever your ask for with these Lego blocks, but pointing out it's unable create its own Lego blocks from scratch has somehow become critically important to point out, as if that makes this all total dead end and it's all a waste of money omg people wake up oh if only they'd listen to me. Why don't people listen to me? Crows are believed to have a theory of mind, and they can count up to 30. I haven't tried it with Claude, but I'm pretty sure it can count at least that high. LLMs are artificial, they're alien, of course they're going to look different. In the analogy where they're simply a next word guesser, one imagines standing at a fridge with a bag of magnetic words, and just pulling a random one from the bag to make ChatGPT. But when you put your hand inside a bag inside a bag inside a bag, twenty times (to represent the dozens of layers in an LLM model), and there are a few hundred million pieces in each bag (for parameters per layer), one imagines that there's a difference; some sort of leap, similar to when life evolved from being a single celled bacterium to a multi-cellular organism. Or maybe we're all just rubes, and some PhD's have conned the world into giving them a bunch of money, because they figured out how to represent essays as a math problem, then wrote some code to solve them, like they did with chess.

dartos 2y ago
There’s a bit of truth to all of what you said. These tools aren’t useless, obviously. But people do really learn hard into confirmation bias and/or personification when it comes to LLMs. I believe it’s entirely because of the term “artificial intelligence” that there is such a divide. If we called them “large statistical language models” instead, nobody would be having this discussion.
2y ago