3 ms·

The reasoning step by step is a good trick to elicit a better answer, but it also seems like really good evidence of GPT-4's lack of actual intelligence, since

by jprete 4y ago

The reasoning step by step is a good trick to elicit a better answer, but it also seems like really good evidence of GPT-4's lack of actual intelligence, since it's clearly not thinking about its answer in any sense. If it were thinking, asking it to go step by step would result in putting the same thoughts to paper, not radically changing them by the act of expressing them in the token stream.

digilypse 4y ago
Don’t humans work similarly? Without an understanding of how to reason through a problem, as children we are likely to give the first intuitive answer that comes to mind. Later we learn techniques of logic and reasoning that allow us to break down a problem into components that can be reasoned about. What this seems to show is that the model does not yet have system-level or architectural guidance on when to employ reasoning, but must be explicitly reminded.
- joshspankit 4y ago
  This has been percolating in my brain too. It seems like a lot of the criticisms of LLMs are actually insights in to how our brains work. The way we’re interacting with GPT for example is starting to feel to me like it has a brain with knowledge structured similarly it ours, but no encapsulating consciousness. The answers it returns then feel like records of the synaptic paths that are connected to the questions. Just like our initial intuitions. I started thinking about this when I saw that visual AI was having trouble “drawing” hands in a way that felt very familiar.
  - jprete 4y ago
    I have had similar thoughts. Generative AI models seem to dream more than think - system-1 thinking - but they are clearly missing system-2 thinking, and/or the thing that tells us to switch systems.
  - simonh 4y ago
    I see it’s knowledge structure as completely different from ours. For example all the GPT variants can give an explanation of how to do arithmetic, or even quite advanced mathematics. They can explain the step by step process. None of them, until quite recently, could actually do it though. The most recent variants can to some extent, but not because they can explain the process. The mechanisms implemented to do maths are completely independent of the mechanisms for explaining it, they are completely unrelated tasks for an LLM. This is because LLMs have been trained on many maths text books and papers explaining maths theory and procedures, so they encode token sequence weightings well suited to generating such texts. That must mean it knows how to do maths, right? I mean it just explained the procedures very clearly, so obviously it can do maths. However maths problems and mathematical expressions are completely different classes of texts from explanatory texts, involving completely unrelated token sequence weightings. In all but the latest GPT variants the token sequence weightings would generally get expressions kind of right, but didn’t understand the significance of numbers hardly at all, so the numeric component of response texts would be basically just made up on the spot. The limitations of probabilistic best guess token sequences just doesn’t work for formal logical structures like maths, so the training of the latest generation models has probably had to be heavily tuned to improve in this area. The implications of this are obvious in the case of mathematics, but it provides a valuable insight into other types of answer. Just because it can explain something, we need to be very careful concluding what that implies it does or doesn’t “know”. Knowledge for us and for LLMs mean completely different things. I’m not at all saying it doesn’t know things, it just knows them in a radically different way from us, that we find hard to understand and reason about, and that can be incredibly counterintuitive to us. If a human can explain how to do something that means they know how to actually do it, but that’s just not at all necessarily so for an LLM. This was blatantly obvious and easy to demonstrate in earlier LLM generations, but is becoming less obvious as workarounds, tuned training texts and calls to specialist models or external APIs are used behind the scenes to close the capability gap between explanatory and practical ability. This is just one example illustrating one of the ways they are fundamentally different from us, but all the cases of LLMs being tricked into generating absurd or weird responses also illustrate many of the other ways their knowledge and reasoning architecture varies enormously from ours. These things are incredibly capable, but are essentially very smart and sophisticated, but also very alien intelligences.
akomtu 4y ago
The sample above gives a cue to how rudimentary reasoning can be done. After GPT answers a question, it asks itself "what are the possible flaws in this answer?" then it evalutes each potential flow, up to 3-5, with another question "does this flaw really apply?" It's going to be like the mcts addition that made alphago so strong.

by jprete 4y ago

digilypse 4y ago

Don’t humans work similarly? Without an understanding of how to reason through a problem, as children we are likely to give the first intuitive answer that comes to mind. Later we learn techniques of logic and reasoning that allow us to break down a problem into components that can be reasoned about. What this seems to show is that the model does not yet have system-level or architectural guidance on when to employ reasoning, but must be explicitly reminded.

joshspankit 4y ago
This has been percolating in my brain too. It seems like a lot of the criticisms of LLMs are actually insights in to how our brains work. The way we’re interacting with GPT for example is starting to feel to me like it has a brain with knowledge structured similarly it ours, but no encapsulating consciousness. The answers it returns then feel like records of the synaptic paths that are connected to the questions. Just like our initial intuitions. I started thinking about this when I saw that visual AI was having trouble “drawing” hands in a way that felt very familiar.
- jprete 4y ago
  I have had similar thoughts. Generative AI models seem to dream more than think - system-1 thinking - but they are clearly missing system-2 thinking, and/or the thing that tells us to switch systems.
- simonh 4y ago
  I see it’s knowledge structure as completely different from ours. For example all the GPT variants can give an explanation of how to do arithmetic, or even quite advanced mathematics. They can explain the step by step process. None of them, until quite recently, could actually do it though. The most recent variants can to some extent, but not because they can explain the process. The mechanisms implemented to do maths are completely independent of the mechanisms for explaining it, they are completely unrelated tasks for an LLM. This is because LLMs have been trained on many maths text books and papers explaining maths theory and procedures, so they encode token sequence weightings well suited to generating such texts. That must mean it knows how to do maths, right? I mean it just explained the procedures very clearly, so obviously it can do maths. However maths problems and mathematical expressions are completely different classes of texts from explanatory texts, involving completely unrelated token sequence weightings. In all but the latest GPT variants the token sequence weightings would generally get expressions kind of right, but didn’t understand the significance of numbers hardly at all, so the numeric component of response texts would be basically just made up on the spot. The limitations of probabilistic best guess token sequences just doesn’t work for formal logical structures like maths, so the training of the latest generation models has probably had to be heavily tuned to improve in this area. The implications of this are obvious in the case of mathematics, but it provides a valuable insight into other types of answer. Just because it can explain something, we need to be very careful concluding what that implies it does or doesn’t “know”. Knowledge for us and for LLMs mean completely different things. I’m not at all saying it doesn’t know things, it just knows them in a radically different way from us, that we find hard to understand and reason about, and that can be incredibly counterintuitive to us. If a human can explain how to do something that means they know how to actually do it, but that’s just not at all necessarily so for an LLM. This was blatantly obvious and easy to demonstrate in earlier LLM generations, but is becoming less obvious as workarounds, tuned training texts and calls to specialist models or external APIs are used behind the scenes to close the capability gap between explanatory and practical ability. This is just one example illustrating one of the ways they are fundamentally different from us, but all the cases of LLMs being tricked into generating absurd or weird responses also illustrate many of the other ways their knowledge and reasoning architecture varies enormously from ours. These things are incredibly capable, but are essentially very smart and sophisticated, but also very alien intelligences.

akomtu 4y ago

The sample above gives a cue to how rudimentary reasoning can be done. After GPT answers a question, it asks itself "what are the possible flaws in this answer?" then it evalutes each potential flow, up to 3-5, with another question "does this flaw really apply?" It's going to be like the mcts addition that made alphago so strong.