24 ms·

LLMs aren’t reasoning about the puzzle. They’re predicting the most likely text to print out, based on the input and the model/training data. If the solution i

• jatins 4y ago
  I think we are past the "just predicting the next token" stage. GPT and it's various incarnations do exhibit behaviour that most people will describe as thinking
  • localplume 4y ago
    thats because people anthropormophize literally anything, and many treat some animals as if they have the same intelligence as humans. GPT has always been just a charade that people mistake for intelligence. Its a glorified text prediction engine with some basic pattern matching.
    • int_19h 4y ago
      "Descartes denied that animals had reason or intelligence. He argued that animals did not lack sensations or perceptions, but these could be explained mechanistically. Whereas humans had a soul, or mind, and were able to feel pain and anxiety, animals by virtue of not having a soul could not feel pain or anxiety. If animals showed signs of distress then this was to protect the body from damage, but the innate state needed for them to suffer was absent."
      • bun_at_work 4y ago
        Your comment brings up the challenge of defining intelligence and sentience, especially with these new LLMs shaking things up, even for HN commenters. It's tough to define these terms in a way that includes only humans and excludes other life forms or even LLMs. This might mean we either made up these concepts, or we're not alone in having these traits. Without a solid definition, how can we say LLMs aren't intelligent? If we make a definition that includes both us and LLMs, would we accept them as intelligent? And could we even exclude ourselves? We need clear definitions to talk about the intelligence and sentience of LLMs, AI, or any life forms. But finding those definitions is hard, and it might clash with our human ego. Discussing these terms without definitions feels like a waste of time. Still, your Descartes reference reminds us that our understanding of human experiences keeps changing, and our current definitions might not be spot-on. (this comment was cleaned up with GPT-4 :D)
    • nuancebydefault 4y ago
      It's a charade, it mimics intelligence. Let's take it ine step further... Suppose it mimics it so well that it becomes indistinguishable for any human from being intelligent. Then still it would not be intelligent, one could argue. But in that case you could also argue that no person is intelligent. The point being, intelligence cannot be defined. And, just maybe, that is the case because intelligence is not a reality, just something we made up.
      • Jeff_Brown 4y ago
        Objective measures of intelligence are easy to come up with. The LSAT is one. (Not a great one -- GPT-4 passes it, after all -- but an objective one.) Consciousness, on the other hand, really might be an illusion.
  • deleted 4y ago
    [deleted]
  • a_wild_dandan 4y ago
    Yeah, calling AI a "token predictor" is like dismissing human cognition dumb "piles of electrical signal transmitters." We don't even understand our minds, let alone what constitutes any mind, be it alien or far simpler than ours. Simple != thoughtless. Different != thoughtless. Less capable != thoughtless. A human black box categorically dismissing all qualia or cognition from another remarkable black box feels so wildly arrogant and anthropocentric. Which, I suppose, is the most historically on-brand behavior for our species.
    • WoodenChair 4y ago
      It might be a black box to you, but it’s not in the same way the human brain is to researchers. We essentially understand how LLMs work. No, we may not reason about individual weights. But in general it is assigning probabilities to different possible next tokens based on their occurrences in the training set and then choosing sometimes the most likely, sometimes a random one, and often one based on additional training from human input (e.g. instruct). It’s not using its neurons to do fundamental logic as the earlier posts in the thread point out. Stephen Wolfram explains this in simple terms.[0] 0: https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/ https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...
      • TchoBeer 4y ago
        >based on their occurrences in the training set the words "based on" are doing a lot of work here. No, we don't know what sort of stuff it learns from its training data nor do we know what sorts of reasoning it does, and the link you sent doesn't disagree.
        • WoodenChair 4y ago
          We know that the relative location of the tokens in the training data influences the relative locations of the predicted tokens. Yes the specifics of any given related tokens are a black box because we're not going to go analyze billions of weights for every token we're interested in. But it's a statistical model, not a logic model.
  • thomastjeffery 4y ago
    Just because GPT exhibits a behavior does not mean it performs that behavior. You are using those weasel words for a very good reason! Language is a symbolic representation of behavior. GPT takes a corpus of example text, tokenizes it, and models the tokens. The model isn't based on any rules: it's entirely implicit. There are no subjects and no logic involved. Any "understanding" that GPT exhibits was present in the text itself, not GPT's model of that text. The reason GPT can find text that "makes sense", instead of text that "didn't make sense", is that GPT's model is a close match for grammar. When people wrote the text in GPT's corpus, they correctly organized "stuff that makes sense" into a string of letters. The person used grammar, symbols, and familiar phrases to model ideas into text. GPT used nothing but the text itself to model the text. GPT organized all the patterns that were present in the corpus text, without ever knowing why those patterns were used.
    • Jeff_Brown 4y ago
      > GPT used nothing but the text itself to model the text. I used nothing but my sensory input to model the world, and yet I have a model of the world, not (just) of sensory input. There is an interesting question, though, of whether information without experience is enough to generate understanding. I doubt it.
      • thomastjeffery 4y ago
        Not true. You used evolution, too. The structure of your brain growth is the result of complex DNA instructions that have been mutated and those mutations filtered over billions of iterations of competition. There are some patterns of thought that are inherent to that structure, and not the result of your own lived experience. For example, you would probably dislike pain with similar responses to your original pain experience; and also similar to my lived pain experiences. Surely, there are some foundational patterns that define our interactions with language.
      • dTal 4y ago
        In what sense is your "experience" (mediated through your senses) more valid than a language model's "experience" of being fed tokens? Token input is just a type of sense, surely?
• valine 4y ago
  How do you know the model isn’t internally reasoning about the problem? It’s a 175B+ parameter model. If, during training, some collection of weights exist along the gradient that approximate cognition, then it’s highly likely the optimizer would select those weights over more specialized memorization weights. It’s also possible, likely even, that the model is capable of both memorization and cognition, and in this case the “memorization neurons” are driving the prediction.
  • albertgoeswoof 4y ago
    How could you prove this?
    • valine 4y ago
      Just spitballing, I think you’d need a benchmark that contains novel logic puzzles, not contained in the training set, that don’t resemble any existing logic puzzles. The problem with the goat question is that the model is falling back on memorized answers. If the model is in fact capable of cognition, you’d have better odds of triggering the ability with problems that are dissimilar to anything in the training set.
    • fancyfredbot 4y ago
      People have shown GPT has an internal model of the state of a game of Othello: Https://arxiv.org/abs/2210.13382
      • pja 4y ago
        More accurately: a GPT derived DNN that’s been specifically trained (or fine-tuned, if you want to use OpenAI’s language) on a dataset of Othello games ends up with an internal model of an Othello board. It looks like OpenAI have specifically added Othello game handling to chat.openai.org, so I guess they’ve done the same fine-tuning to ChatGPT? It would be interesting to know how good an untuned GPT3/4 was at Othello & whether OpenAI has fine-tuned it or not! (Having just tried a few moves, it looks like ChatGPT is just as bad at Othello as it was at chess, so it’s interesting that it knows the initial board layout but can’t actually play any moves correctly: Every updated board it prints out is completely wrong.)
• LeanderK 4y ago
  at this stages ranting about assigning probabilities is not reasoning is just dismissive. Mentioning its predictive character doesn't prove anything. We reason and make mistake too, even if I think really hard about a problem I can still make an mistake in my reasoning. And the ever occurring reference to training data just completely ignores generalisation. ChatGPT is not memorising the dataset, we have known this for years with more trivial neural network. Generalisation capabilities of neural network has been the subject of intense study for years. The idea that we are just mapping it to samples occurring in the dataset is just ignoring the entire field of statistical learning.
  • albertgoeswoof 4y ago
    Sorry but this is the reason it’s unable to solve the parents puzzle. It’s doing a lot but it’s not logically reasoning about the puzzle, and in this case it’s not exhibiting logical behaviour in the result so it’s really obvious to see. Eg when solving this puzzle you might visualise the lion/goat/cabbage, and walk through the scenarios in your head back and forth multiple times until you find a solution that works. A LLM won’t solve it like this. You could ask it to, and it will list out the scenarios of how it might do it, but it’s essentially an illusion of logical reasoning.
    • throwwwaway69 4y ago
      Trying to claim you definitively know why it didn't solve the parent's puzzle is virtually impossible. There are way too many factors and nothing here is obvious. Your claims just reinforce that you don't really know what you're talking about.
    • int_19h 4y ago
      If you gave this puzzle to a human, I bet that a non-insignificant proportion would respond to it as if it were the traditional puzzle as soon as they hear words "cabbage", "lion", and "goat". It's not exactly surprising that a model trained on human outputs would make the same assumption. But that doesn't mean that it can't reason about it properly if you point out that the assumption was incorrect. With Bing, you don't even need to tell you what it assumed wrong - I just told it that it's not quite the same as the classic puzzle, and it responded by correctly identifying the difference and asking me if that's what I meant, but forgot that lion still eats the goat. When I pointed that out, it solved the puzzle correctly. Generally speaking, I think your point that "when solving the puzzle you might visualize" is correct, but that is orthogonal to the ability of LLM to reason in general. Rather, it has a hard time to reason about things it doesn't understand well enough (i.e. the ones for which its internal model that was built up by training is in is way off). This seems to be generally the case for anything having to do with spatial orientation - even fairly simple multi-step tasks involving concepts like "left" vs "right" or "on this side" vs "on that side" can get hilariously wrong. But if you give it a different task, you can see reasoning in action. For example, have it play guess-the-animal game with you while telling it to "think out loud".
• Eji1700 4y ago
  It is disheartening to see how many people are trying to tell you you're wrong when this is literally what it does. It's a very powerful and useful feature, but the over selling of AI has led to people who just want this to be so much more than it actually is. It sees goat, lion, cabbage, and looks for something that said goat/lion/cabbage. It does not have a concept of "leave alone" and it's not assigning entities with parameters to each item. It does care about things like sentence structure and what not, so it's more complex than a basic lookup, but the amount of borderline worship this is getting is disturbing.
  • baq 4y ago
    The problem with this simplification is a bog standard Markov chain fits the description as well, but quality of predictions is rather different. Yes the LLM does generate text. No it doesn’t ‘just generate text that’s it’.
    • peterashford 4y ago
      "It generates text better than a Markov chain" - problem solved
      • baq 4y ago
        Classic goal post moving.
        • peterashford 4y ago
          Not really, I think the original post was just being a post, not a scientific paper. Sometimes people speak normally
    • Izkata 4y ago
      The biggest problem I've seen when people try to explain it is in the other direction, not people describing something generic that can be interpreted as a Markov chain, they're actually describing a Markov chain without realizing it. Literally "it predicts word-by-word using the most likely next word".
• deleted 4y ago
  [deleted]
• red75prime 4y ago
  > If the solution is logical but unlikely The likeliness of the solution depends on context. If context is, say, a textbook on logical puzzles, then the probability of the logical solution is high. If an LLM fails to reflect it, then it isn't good enough at predicting the text. Yes, it could be possible that the required size of the model and training data to make it solve such puzzles consistently is impractical (or outright unachievable in principle). But the model being "just a text predictor" has nothing to do with that impossibility.
• kromem 4y ago
  You are incorrect and it's really time for this misinformation to die out before it perpetuates misuse from misunderstanding model capabilities. The Othello GPT research from Harvard months ago demonstrated that even a simple GPT model is capable of building world representations from which it reasons outputs. This makes intuitive sense if you understand the training, as where possible having reversed an abstraction in the NN is going to perform better than simply extrapolating predictively from the data. Not only is GPT-4 more robust at logic puzzles its predecessor failed, I've seen it solve unique riddles outside any training data and the paper has explicit examples of critical reasoning, especially in the appendix. It is extremely unlikely given the Harvard research and the size of the training data and NN that there isn't some degree of specialized critical reasoning which has developed in the NN. The emerging challenge for researchers moving forward is to get better insight into the black box and where these capabilities have developed and where it's still falling into just a fancy Markov chain. But comments like yours reflect an increasingly obsolete and yet increasingly popular misinformation online around the way they operate. So someone reading your comment might not think to do things like what the Bing team added with providing an internal monologue for reasoning, or guiding it towards extended chain of thought reasoning, because they would be engaging with the models thinking it's only frequency based context relative to the training set that matters. If you haven't engaged with emerging research from the past year, you may want to brush up on your reading.
• zeofig 4y ago
  Word. There is no other way it can be. Not to say these "AI"s aren't useful and impressive, but they have limitations.
• bulbosaur123 4y ago
  > LLMs aren’t reasoning about the puzzle. They’re predicting the most likely text to print out, based on the input and the model/training data. Just like you.
  • xdennis 4y ago
    When albertgoeswoof reasons about a puzzle he models the actual actions in his head. He uses logic and visualization to arrive at the solution, not language. He then uses language to output the solution, or says he doesn't know if he fails. When LLMs are presented with a problem they search for a solution based on the language model. And when they can't find a solution, there's always a match for something that looks like a solution.
    • astrange 4y ago
      I'm reminded of the interview where a researcher asks firemen how they make decisions under pressure, and the fireman answers that he never makes any decisions. Or in other words, people can use implicit logic to solve puzzles. Similarly LLMs can implicitly be fine-tuned into logic models by asking them to solve a puzzle, insofar as that logic model fits in their weights. Transformers are very flexible that way.
• JyB 4y ago
  https://arxiv.org/pdf/2210.13382.pdf https://arxiv.org/pdf/2210.13382.pdf