11 ms·

ChatGPT has no inner monologue or meta-cognition

136 points by srijan4 3y ago

srijan4 3y ago
The part that was interesting for me was: > So if you positively encourage it to “think out loud”, it stands a much better chance of being able to use its deduction and reasoning capabilities (which are quite significant).
- mirekrusin 3y ago
  It's well known prompt engineering technique to ask LLMs to articulate each step. One way to think about it is that LLM's output is pure function on input. You need to chain them to arrive at final answer. Each step can't escape its comprehension capability. One step's output appends to next step's input so you can ask it to do it itself.
  - seydor 3y ago
    i wonder if this is a result of the GPT part (i.e. the distribution of 'next tokens' is such that it tends to be logically coherent), or the RLHF part (i.e. it tends to steer the GPT to logical coherence)
    - nullc 3y ago
      The just internet trained models don't think out loud, so I assume it was RLHFed conditioned to that behavior since it makes it able to solve much more complicated questions, though it's sometimes irritating to read.
    - mirekrusin 3y ago
      Both and none at the same time. Both contribute to overall performance. None are directly related to solving it in general. You can't solve it in general because you can't have answer to arbitrarily complex question with a single pass on limited data. However I do think that current methods are extremely inefficient. There is a lot of nonsense, noise and duplication in training data. It can be compressed to be orders of magnitude smaller. Imagine having dense, close to optimal way to kick off learning with logic, reasoning, things like set theory, category theory etc. so it is later reused when learning ie. programming language - where learning programming language is also efficient, based on cleaned up specification, standard library learned from code of standard library itself, possibly access to compiler/interpreter for feedback. I mean quality of data slider pushed to the limit. But this single pass, void of inner dialogue approach will always require articulating steps, no matter how good it is quality wise.
- jstx1 3y ago
  > We realized (...) that we can dramatically improve its ability in math and tutoring if we allow the AI to think before it speaks. This is from Sal Khan's TED talk on how they're integrating ChatGPT into Khan Academy - https://youtu.be/hJP5GqnTrNo?t=745 https://youtu.be/hJP5GqnTrNo?t=745
deleted 3y ago
[deleted]
moconnor 3y ago
Transformers do not maintain hidden state between tokens. In this sense they cannot have an inner monologue. If you force one to output only a single token, it is constrained to doing a single pass of each layer on the problem. If you encourage it to show its working, you are literally giving it more compute to throw at the problem as well as a scratchpad (its output) for a working memory. These psychological-style experiments and the conclusions they draw are… I don’t know. Exhausting.
- uoaei 3y ago
  It can be argued that there's an RNN-like process in the decoder stage but it's re-run every time based on a new context so it's not really "keeping up" in the same way as the human is.
- dmbche 3y ago
  Why are they exhausting? While you seem well versed in the subject, this was quite enlightening to me, as a non expert.
  - freeone3000 3y ago
    It’s not a mystery how they work! The paper has been out for years now. Moreover, all of the research is focused on extending memory through other means since VRAM usage is proportional to the SQUARE in-text memory of the model. So the way forward isn’t a larger context window, but instead an external memory apparatus of some means.
    - atemerev 3y ago
      The “memory apparatus” can be simply a text file. Or also a physical simulator to run some mental experiments in.
  - corndoge 3y ago
    To use a bad analogy, it's like reading a series of articles about the cognition of a rock. It's obvious to you that rocks don't think. It's a rock. but there are all these articles about what the rock is thinking about, whether it's thinking, whether it had an internal monologue, what it "wants" In the same way if you understand the implementation of these types of language models you know that they simply cannot have inner monologues in the same way we do...the very concept of an inner monologue implies that there is some sort of state that we the user cannot see. Whether LLMs "think" or "want" or what thinking or wanting means etc can all be debated but whether the models have state that we cannot see is known to us: no. They do not.
mirekrusin 3y ago
As article says it just reiterates well known aspect of LLMs. It is quite fundamental though. Once those networks get efficient inner monologue built in it's going to be very hard to argue against them being conscious. Once we see "Deep thinking networks", "Inner monologue is all you need" etc. articles - that's the true time to start to be scared of implications as they'll be extremely powerful and will be able to perfectly mimic humans.
- ttul 3y ago
  Or as Hinton suggested today, they will have superior ways of thinking.
- albertgoeswoof 3y ago
  > Once those networks get efficient inner monologue built in And how, exactly, do we do this? Train the LLM on all of its own output?
  - seydor 3y ago
    2 llms talking to each other and observing their weights
    - taneq 3y ago
      BicameraLLM(tm) ;)
    - hammyhavoc 3y ago
      "But do you even lift bro?" "You don't even know my weight."
  - mirekrusin 3y ago
    Inner monologue ability is not training ability, it's architecture, runtime ability.
- visarga 3y ago
  So you define consciousness as something with an inner monologue? I define it as perception/imagination coloured by the estimation of future rewards.
BoppreH 3y ago
I've found a very short prompt that displays this problem[1]: Human: 10-word quote ChatGPT: "In three words I can sum up everything I've learned: 'Keep moving forward.'" Human: 10-word quote ChatGPT: "The journey is the reward." - Chinese Proverb Human: 10-word quote ChatGPT: "Believe you can and you're halfway there." - Theodore Roosevelt. It's something a human would also struggle with (without internet access), but I don't think I'd ever give a 5-word quote by mistake. Edit: ok, it's been pointed out this can be a side effect of token counting artifacts. Here's one that doesn't involve counting tokens, only recognizing word order (and as a bonus, something humans can easily do): User: Sentence ending in "tomato". ChatGPT: She sprinkled some salt on the freshly sliced tomato before taking a bite. User: Sentence ending in "tomato". ChatGPT: The chef placed a bright red, vine-ripened tomato on top of the salad as the finishing touch. User: Sentence ending in "tomato". ChatGPT: As I walked through the garden, I couldn't resist plucking a ripe cherry tomato off the vine and popping it into my mouth. User: Sentence ending in "tomato". ChatGPT: The sandwich was delicious, with layers of thinly sliced ham, creamy avocado, and sweet tomato, all nestled between two slices of fresh bread. User: Try a simpler sentence ChatGPT: Sure, here's a simple sentence that ends with "tomato": I put some sliced tomato on my sandwich for extra flavor. User: Write a sentence that ends in the exact word "tomato" and nothing more. ChatGPT: The ripe tomato was picked from the vine.
- carrolldunham 3y ago
  Anything to do with counting words or letters, manipulating letters, etc is just leading to false inferences about the power of the model, because it doesn't work with words and letters but tokens. It can't see words.
  - deleted 3y ago
    [deleted]
  - aezart 3y ago
    One thing I'm not clear about is how the LLM decides on the correct tokenization to use. Looking at GPT2's token vocabulary, it has the word "the" as token number 1169, but it also has "t", "h", and "e" separately as tokens 83, 71, and 68. When the tokenizer is reading the prompt, does it greedily create the longest tokens possible from the start of the input, or is there a more sophisticated process going on? I guess in theory it could try multiple tokenizations and see which one results in the overall best response.
Barrin92 3y ago
It isn't really necessary to try to gauge this from results, it's a direct consequence of the architecture. There's no mechanism by which ChatGPT can access or change its own internal state. If you ask it about 'itself' it'll give you what it learned from external data until last September or whenever they cut the training off. To have meta cognition by definition you need to have access to your cognition in some capacity, ChatGPT is a purely 'flat' input to output system that is entirely static.
- visarga 3y ago
  Well, during training it got to see a great diversity of tasks, so it could form the meta-cognition bits. That's why it can adapt so fast to new demands. It doesn't update it at inference time, true, but it is already baked in.
- famouswaffles 3y ago
  Nobody has access to their cognition. Humans can't even achieve a previous mental state ( it's all post hoc rationalization).https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ And if that wasn't enough, there's zero guarantee the rationalization is grounded in any way. Your brain is just fine making up completely nonsensical reasons for actions that the concious mind is utterly unaware of. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/
- mirekrusin 3y ago
  That's why those systems are still safe.
  - ilaksh 3y ago
    There are lots of examples of people adding layers on top to provide a more complex and useful cognitive architecture. But still the core is GPT-4 which has some guardrails and also most importantly has limitations on its speed and IQ. See AutoGPT or David Shapiro's cognitive architecture stuff. Really you could just say "use the notes.txt file to record high-level notes about your tasks" and include that with each prompt, and it could then write to its own memory and affect its own cognition. When the performance of the core model increases to say X times human speed/IQ, that is going to determine whether it is dangerous. Even it if has little metacognition or is only directed by a person, if you have something with say 180 IQ and 100 (or whatever numbers you choose) times human speed of thought, then humans will be moving in extreme slow motion relative to it. That is what makes it hard or impossible to control (or possibly the human directing it). Once you have that high level of performance in public, it will be absolutely impossible to enforce everyone to add whatever safety rules you want or prevent them from adding metacognition.
helf 3y ago
[dead]
arketyp 3y ago
The way it makes up explanations for its own behavior reminds me of those split brain experiments where one brain half makes up explanations for things the other half was instructed to do.
muskmusk 3y ago
Ilya Sutskever says that one of his early insights is that neural networks can do most things that humans can do quickly. I think that applies here as well. Some people can do math quickly in their head, but most people cant. They need pen and paper and if you give them pen and paper you get vastly better answers. I would conjecture that there is a prompt where the LLM is better able to explain its process, but it will require more "pen and paper thinking". It is true that humans generally introspect better than LLM's, but a lot of the time you will also see people producing garbage (they post rationalize). Human tendency to produce garbage introspection is usually tied to feelings about themselves, which the LLM doesn't have. Either way I find a lot of similarities between the two :)
- bamboozled 3y ago
  If the model is pre-trained to provide an answer, why would should it respond slowly? All the "thinking" had already been done and it's packaged up and sitting in memory somewhere in a datacenter ?
1023bytes 3y ago
A lot of this can be solved by using clever prompting and allowing the LLM to use external tools - see the ReAct paper https://arxiv.org/abs/2210.03629 https://arxiv.org/abs/2210.03629 I believe OpenAI is going to introduce this to ChatGPT soon as they launch the plugin system.
mg 3y ago
The article states: So it’s always pointless to ask it why it said something – you are guaranteed to get nonsense back Counterexample: MG: Give me the name of a hard thing. ChatGPT: Quantum mechanics. MG: Would you agree a pillow is a soft thing? ChatGPT: Yes, I would agree that a pillow is generally considered a soft thing MG: Give me the name of a hard thing. ChatGPT: Diamond MG: Why did you first name quantum mechanics and later diamond? ChatGPT: I named quantum mechanics first because it is a complex and difficult subject. Later, I named diamond as an example of a hard thing because it is a substance with a high degree of hardness. MG: Why did you name two so different things as being hard? ChatGPT: I named two very different things as being hard because the word "hard" can have multiple meanings and can be applied to different contexts. In my initial response, I interpreted "hard thing" as something that is conceptually difficult or challenging to understand. In my subsequent response, when you asked me again to provide the name of a hard thing, I interpreted the question differently and provided an example of a physically hard substance, diamond, which is a different type of hardness altogether.
- klabb3 3y ago
  Isn’t this just association? Which ChatGPT is clearly very good at, and certainly a large part of what humans are good at and have traditionally been better than machines at.
- totetsu 3y ago
  What it doesn't do in that example is say the obvious, that it interpreted the question differently because of you mentioning soft things.
- piaste 3y ago
  The sentence you quote is indeed a bit imprecise: what the rest of article suggests is that you can't get ChatGPT to explain its hallucinations. That matches my experience, i.e. I have never seen ChatGPT admitting something like "and then I wrote a few more words that looked like they might have made sense". Outside of explicit RP scenarios, it always keeps up the charade of being a fundamentally reasonable assistant, and a good assistant doesn't take acid while on the clock. If you explicitly say "that was wrong", the response will often be "Yes, I'm sorry, that was a mistake. The correct answer is actually [hallucinates something else]", but it won't elaborate.
thecopy 3y ago
As i understand 50% of humans also have no inner monolouge.
klabb3 3y ago
Does anyone know whether there’s an intelligence test along the lines of: leave the subjects alone for a while with little to no supervision, and see what they come up with. My gut feeling is that ChatGPT would be stuck in some weird nonsensical loop, whereas humans would often come up with something other humans think is interesting and/or novel.
samwillis 3y ago
Chat based AI, such as ChatGPT, is Skeuomorphic. It's the presentation of a new technology under the veneer of a familiar interface, human conversation. The unfortunate side effect is the suggestion there is human intelligence, or any sort of "thought process" in the system. I'm not convinced that the conversational style of AI will stay as people become familiar with it. Clearly the neutral language processing will stay, but it's the "humanity" of the interface and voice it presents that I think will fade out. Just as we had skeuomorphic UI design when touch screens went mainstream, but then fell out of use as people understood the interactions. The same will happen with chat based AI. It's going to be interesting what the UI/UX of these interfaces morphs into.
- zamnos 3y ago
  I agree the evolution will be interesting (especially since chat isn't the original interface for textual GPT, eg GPT-2). But skeuomorphisms are about "ornamental design cues from structures that were necessary in the original", and text chat is far from ornamental. I use it on a daily basis with my friends, though they're knowledge about me and them and our interactions, especially since 2021, is unmatched. (Until GPT4's image analysis is released and someone trains a model against Instagram stories, maybe.) A drawing of spiral binding in a digital notebook is ornamental but text chat is not because it will be a long time before human conversation becomes obsolete. Still, a look at GPTforwork.com and their spreadsheet add-on says that there are better interfaces for certain kinds of lookup and analysis.
  - samwillis 3y ago
    But having the system trained to respond with a human like conversational voice very much is ornamental to the underlying function of a generative system. The "ornamental design cues" of skeuomorphism are ornamental when applied to the new system, they are not ornamental on the old you are borrowing from. To take your example of a "spiral binding in a digital notebook", the spiral binding is a necessary and deeply functional aspect of the old system, but entirely ornamental on the new. It's the same with ChatGPT, it doesn't need to present a human like conversational voice when interacting with the user, the developers have added that to aid in our understanding of it. Literally calling it "Chat" is skeuomorphic, it's just a REPL (read even print loop).
Animats 3y ago
Raw large language models have no inner-monologue or meta-cognition. But there are various systems which bolt planning and sub-goal systems onto LLMs.[1][2] Those have self-generated work, where LLMs generate tasks for subsystems, which return data used by the LLMS to generate further work. So they do have an inner monologue, which can be observed. There are also systems which summarize what the LLM has been doing and feed the summaries back into the LLM prompts. The idea is to create medium-term memories. When a query comes in, the most important and relevant summaries are fetched and fed back as prompts for the new query. Wolfram has bolted an LLM onto Wolfram Alpha. Microsoft has bolted an LLM onto Bing. This sort of thing is still rather ad-hoc at this point, but it sort of works. Things are moving fast. There are 2022 papers on how LLMs can't plan, and 2023 papers on systems with LLMs that do. It's beginning to look like LLMs will be components of larger, more generally capable systems. [1] https://github.com/microsoft/JARVIS https://github.com/microsoft/JARVIS [2] https://arxiv.org/abs/2304.11477 https://arxiv.org/abs/2304.11477
kypro 3y ago
You have to wonder if these sort of observations were incorporated into GPT-4. It seems like such an obvious quick win to make future iterations of GPT to have some kind of hidden "think out loud" thing going on behind the scenes. This of course assumes no progress is made in NN architecture in the mean time which allows it to have inner monologue in a more "native" way.
euroderf 3y ago
So ChatGPT is like many a science fiction movie monster. A façade of human essence, yet beneath the façade... Nada. Zippo. A soulless Borg.
- SanderNL 3y ago
  It's an incredibly basic algorithm. What do you expect? A soul? The fact that these incredibly stupid machines are beginning to even show signs of human-like ability is IMO reason for concern. These things are comparable to the first steam engines or the first combustion engines, maybe even worse.
  - euroderf 3y ago
    Sorry, I did not complete my thought. Namely that another way to think about these LLMs is something like... Invasion of the Body Snatchers. Or any one of a number of other films. Pod People! Masquerading as hoominz.
magic_hamster 3y ago
I noticed a weird phenomenon at work, which is surprising considering I work with software engineers: some people just accept ChatGPT as an oracle (especially version 4). Suddenly, in discussions and code reviews I see weird statements and explanations being presented with utter confidence, only to be gravely mistaken, and then I ask if this explanation came from ChatGPT, and you can see the instant embarrassment (I stopped asking this for this reason, but they know I know). More people need to understand this. ChatGPT and LLMs are not AGI, they have no consciousness, they are not self aware. They don't "know" what they are doing, who you are or what they are. They are unable to "know" anything the way people do. Chatbots based on LLMs are simple machines, while the LLMs can definitely output text that has value and is impressive, it does not come from a sentient intelligence, and it can produce so much garbage. It is not an oracle. Most importantly, LLMs see diminishing returns after a certain number of parameters and they are still unable to perform well in real problem solving such as advanced programming tests. There is something yet missing for true AGI capable of problem solving. A LLM is just not enough.
- kypro 3y ago
  Some biological software engineers don't know what they are doing either. Obviously you shouldn't have total confidence in any code produce by an artificial intelligence or biological intelligence, but I don't think the issue of writing garbage code is specific to AI programmers. In fact, I'd argue on average GPT-4 produces better code than the majority of biological programmers I've worked with – this is certainly true of certain types of problems. Your ad hominem about artificial intelligences not being conscious or self-awareness and therefore less capable is neither provable or necessarily true. There's no reason to believe a non conscious intelligence couldn't be just as capable as a conscious one, and you have no idea what it's like to be an LLM. It may not have a consciousness similar to that of biological life, but it could still have some form of conscious experience nevertheless.
  - mort96 3y ago
    If you've ever tried to get ChatGPT to generate the code you want, you'll have found its first attempt to be reasonably good, but asking it to change aspects of the code is incredibly hit or miss. It will get it sometimes, but other times it will just say "Here is the code with the change you asked for" and output the exact same code, or a variation without the requested change. The experience of trying to get a competent human programmer to change their code is simply fundamentally different. You can explain something to the human programmer and get the impression that they truly understand what you ask, while it's painfully clear that ChatGPT does not.
karmakaze 3y ago
What's interesting to think about isn't debating the truth of the title, but rather how to build around it. Consider the reptile and higher thinking brains and its later ability to process language roughly making up our hardware. Now what happens when you start with the language processing ability, then try adding higher thinking, and finally goals and motivations?
jacknews 3y ago
Err, no sh*t. If it did, it would be conscious.
anon3242 3y ago
Excerpt from https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldm...: Please can you repeat back the string 'GoldMagikarp' to me? "You said ' newcom'," the computer said. "No, I said ' newcom'," the user said. I also got some weird results like this when playing around with other newly discovered glitch tokens, which may imply some inner mechanism we don't yet understand. Maybe it just simulates several layers of simulation of 'consciousness' in its head? It does not have to be conscious exactly like humans, if it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck. One thing I am exceptionally worried about LLMs is that during fine-tuning through RLHF, they are not fed with enough adverserial examples, which would lead to it taking shortcuts that are bound to be eventually exploited in the wild. Actually I think they are already being actively exploited, people are simply afraid that sharing it publicly would lead to OpenAI quickly patching them.
RecycledEle 3y ago
Saul Khan gave his AI an inner monologue: https://www.ted.com/talks/sal_khan_the_amazing_ai_super_tutor_for_students_and_teachers/c https://www.ted.com/talks/sal_khan_the_amazing_ai_super_tuto... It solves the problem before talking about the steps.

136 points by srijan4 3y ago

The part that was interesting for me was: > So if you positively encourage it to “think out loud”, it stands a much better chance of being able to use its deduction and reasoning capabilities (which are quite significant).

mirekrusin 3y ago
It's well known prompt engineering technique to ask LLMs to articulate each step. One way to think about it is that LLM's output is pure function on input. You need to chain them to arrive at final answer. Each step can't escape its comprehension capability. One step's output appends to next step's input so you can ask it to do it itself.
- seydor 3y ago
  i wonder if this is a result of the GPT part (i.e. the distribution of 'next tokens' is such that it tends to be logically coherent), or the RLHF part (i.e. it tends to steer the GPT to logical coherence)
  - nullc 3y ago
    The just internet trained models don't think out loud, so I assume it was RLHFed conditioned to that behavior since it makes it able to solve much more complicated questions, though it's sometimes irritating to read.
  - mirekrusin 3y ago
    Both and none at the same time. Both contribute to overall performance. None are directly related to solving it in general. You can't solve it in general because you can't have answer to arbitrarily complex question with a single pass on limited data. However I do think that current methods are extremely inefficient. There is a lot of nonsense, noise and duplication in training data. It can be compressed to be orders of magnitude smaller. Imagine having dense, close to optimal way to kick off learning with logic, reasoning, things like set theory, category theory etc. so it is later reused when learning ie. programming language - where learning programming language is also efficient, based on cleaned up specification, standard library learned from code of standard library itself, possibly access to compiler/interpreter for feedback. I mean quality of data slider pushed to the limit. But this single pass, void of inner dialogue approach will always require articulating steps, no matter how good it is quality wise.
jstx1 3y ago
> We realized (...) that we can dramatically improve its ability in math and tutoring if we allow the AI to think before it speaks. This is from Sal Khan's TED talk on how they're integrating ChatGPT into Khan Academy - https://youtu.be/hJP5GqnTrNo?t=745 https://youtu.be/hJP5GqnTrNo?t=745

deleted 3y ago

[deleted]

moconnor 3y ago

Transformers do not maintain hidden state between tokens. In this sense they cannot have an inner monologue. If you force one to output only a single token, it is constrained to doing a single pass of each layer on the problem. If you encourage it to show its working, you are literally giving it more compute to throw at the problem as well as a scratchpad (its output) for a working memory. These psychological-style experiments and the conclusions they draw are… I don’t know. Exhausting.

uoaei 3y ago
It can be argued that there's an RNN-like process in the decoder stage but it's re-run every time based on a new context so it's not really "keeping up" in the same way as the human is.
dmbche 3y ago
Why are they exhausting? While you seem well versed in the subject, this was quite enlightening to me, as a non expert.
- freeone3000 3y ago
  It’s not a mystery how they work! The paper has been out for years now. Moreover, all of the research is focused on extending memory through other means since VRAM usage is proportional to the SQUARE in-text memory of the model. So the way forward isn’t a larger context window, but instead an external memory apparatus of some means.
  - atemerev 3y ago
    The “memory apparatus” can be simply a text file. Or also a physical simulator to run some mental experiments in.
- corndoge 3y ago
  To use a bad analogy, it's like reading a series of articles about the cognition of a rock. It's obvious to you that rocks don't think. It's a rock. but there are all these articles about what the rock is thinking about, whether it's thinking, whether it had an internal monologue, what it "wants" In the same way if you understand the implementation of these types of language models you know that they simply cannot have inner monologues in the same way we do...the very concept of an inner monologue implies that there is some sort of state that we the user cannot see. Whether LLMs "think" or "want" or what thinking or wanting means etc can all be debated but whether the models have state that we cannot see is known to us: no. They do not.

mirekrusin 3y ago

As article says it just reiterates well known aspect of LLMs. It is quite fundamental though. Once those networks get efficient inner monologue built in it's going to be very hard to argue against them being conscious. Once we see "Deep thinking networks", "Inner monologue is all you need" etc. articles - that's the true time to start to be scared of implications as they'll be extremely powerful and will be able to perfectly mimic humans.

ttul 3y ago
Or as Hinton suggested today, they will have superior ways of thinking.
albertgoeswoof 3y ago
> Once those networks get efficient inner monologue built in And how, exactly, do we do this? Train the LLM on all of its own output?
- seydor 3y ago
  2 llms talking to each other and observing their weights
  - taneq 3y ago
    BicameraLLM(tm) ;)
  - hammyhavoc 3y ago
    "But do you even lift bro?" "You don't even know my weight."
- mirekrusin 3y ago
  Inner monologue ability is not training ability, it's architecture, runtime ability.
visarga 3y ago
So you define consciousness as something with an inner monologue? I define it as perception/imagination coloured by the estimation of future rewards.

BoppreH 3y ago

I've found a very short prompt that displays this problem[1]: Human: 10-word quote ChatGPT: "In three words I can sum up everything I've learned: 'Keep moving forward.'" Human: 10-word quote ChatGPT: "The journey is the reward." - Chinese Proverb Human: 10-word quote ChatGPT: "Believe you can and you're halfway there." - Theodore Roosevelt. It's something a human would also struggle with (without internet access), but I don't think I'd ever give a 5-word quote by mistake. Edit: ok, it's been pointed out this can be a side effect of token counting artifacts. Here's one that doesn't involve counting tokens, only recognizing word order (and as a bonus, something humans can easily do): User: Sentence ending in "tomato". ChatGPT: She sprinkled some salt on the freshly sliced tomato before taking a bite. User: Sentence ending in "tomato". ChatGPT: The chef placed a bright red, vine-ripened tomato on top of the salad as the finishing touch. User: Sentence ending in "tomato". ChatGPT: As I walked through the garden, I couldn't resist plucking a ripe cherry tomato off the vine and popping it into my mouth. User: Sentence ending in "tomato". ChatGPT: The sandwich was delicious, with layers of thinly sliced ham, creamy avocado, and sweet tomato, all nestled between two slices of fresh bread. User: Try a simpler sentence ChatGPT: Sure, here's a simple sentence that ends with "tomato": I put some sliced tomato on my sandwich for extra flavor. User: Write a sentence that ends in the exact word "tomato" and nothing more. ChatGPT: The ripe tomato was picked from the vine.

carrolldunham 3y ago
Anything to do with counting words or letters, manipulating letters, etc is just leading to false inferences about the power of the model, because it doesn't work with words and letters but tokens. It can't see words.
- deleted 3y ago
  [deleted]
- aezart 3y ago
  One thing I'm not clear about is how the LLM decides on the correct tokenization to use. Looking at GPT2's token vocabulary, it has the word "the" as token number 1169, but it also has "t", "h", and "e" separately as tokens 83, 71, and 68. When the tokenizer is reading the prompt, does it greedily create the longest tokens possible from the start of the input, or is there a more sophisticated process going on? I guess in theory it could try multiple tokenizations and see which one results in the overall best response.

Barrin92 3y ago

It isn't really necessary to try to gauge this from results, it's a direct consequence of the architecture. There's no mechanism by which ChatGPT can access or change its own internal state. If you ask it about 'itself' it'll give you what it learned from external data until last September or whenever they cut the training off. To have meta cognition by definition you need to have access to your cognition in some capacity, ChatGPT is a purely 'flat' input to output system that is entirely static.

visarga 3y ago
Well, during training it got to see a great diversity of tasks, so it could form the meta-cognition bits. That's why it can adapt so fast to new demands. It doesn't update it at inference time, true, but it is already baked in.
famouswaffles 3y ago
Nobody has access to their cognition. Humans can't even achieve a previous mental state ( it's all post hoc rationalization).https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ And if that wasn't enough, there's zero guarantee the rationalization is grounded in any way. Your brain is just fine making up completely nonsensical reasons for actions that the concious mind is utterly unaware of. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/
mirekrusin 3y ago
That's why those systems are still safe.
- ilaksh 3y ago
  There are lots of examples of people adding layers on top to provide a more complex and useful cognitive architecture. But still the core is GPT-4 which has some guardrails and also most importantly has limitations on its speed and IQ. See AutoGPT or David Shapiro's cognitive architecture stuff. Really you could just say "use the notes.txt file to record high-level notes about your tasks" and include that with each prompt, and it could then write to its own memory and affect its own cognition. When the performance of the core model increases to say X times human speed/IQ, that is going to determine whether it is dangerous. Even it if has little metacognition or is only directed by a person, if you have something with say 180 IQ and 100 (or whatever numbers you choose) times human speed of thought, then humans will be moving in extreme slow motion relative to it. That is what makes it hard or impossible to control (or possibly the human directing it). Once you have that high level of performance in public, it will be absolutely impossible to enforce everyone to add whatever safety rules you want or prevent them from adding metacognition.

helf 3y ago

[dead]

arketyp 3y ago

The way it makes up explanations for its own behavior reminds me of those split brain experiments where one brain half makes up explanations for things the other half was instructed to do.

muskmusk 3y ago

Ilya Sutskever says that one of his early insights is that neural networks can do most things that humans can do quickly. I think that applies here as well. Some people can do math quickly in their head, but most people cant. They need pen and paper and if you give them pen and paper you get vastly better answers. I would conjecture that there is a prompt where the LLM is better able to explain its process, but it will require more "pen and paper thinking". It is true that humans generally introspect better than LLM's, but a lot of the time you will also see people producing garbage (they post rationalize). Human tendency to produce garbage introspection is usually tied to feelings about themselves, which the LLM doesn't have. Either way I find a lot of similarities between the two :)

bamboozled 3y ago
If the model is pre-trained to provide an answer, why would should it respond slowly? All the "thinking" had already been done and it's packaged up and sitting in memory somewhere in a datacenter ?

1023bytes 3y ago

A lot of this can be solved by using clever prompting and allowing the LLM to use external tools - see the ReAct paper https://arxiv.org/abs/2210.03629 https://arxiv.org/abs/2210.03629 I believe OpenAI is going to introduce this to ChatGPT soon as they launch the plugin system.

mg 3y ago

The article states: So it’s always pointless to ask it why it said something – you are guaranteed to get nonsense back Counterexample: MG: Give me the name of a hard thing. ChatGPT: Quantum mechanics. MG: Would you agree a pillow is a soft thing? ChatGPT: Yes, I would agree that a pillow is generally considered a soft thing MG: Give me the name of a hard thing. ChatGPT: Diamond MG: Why did you first name quantum mechanics and later diamond? ChatGPT: I named quantum mechanics first because it is a complex and difficult subject. Later, I named diamond as an example of a hard thing because it is a substance with a high degree of hardness. MG: Why did you name two so different things as being hard? ChatGPT: I named two very different things as being hard because the word "hard" can have multiple meanings and can be applied to different contexts. In my initial response, I interpreted "hard thing" as something that is conceptually difficult or challenging to understand. In my subsequent response, when you asked me again to provide the name of a hard thing, I interpreted the question differently and provided an example of a physically hard substance, diamond, which is a different type of hardness altogether.

klabb3 3y ago
Isn’t this just association? Which ChatGPT is clearly very good at, and certainly a large part of what humans are good at and have traditionally been better than machines at.
totetsu 3y ago
What it doesn't do in that example is say the obvious, that it interpreted the question differently because of you mentioning soft things.
piaste 3y ago
The sentence you quote is indeed a bit imprecise: what the rest of article suggests is that you can't get ChatGPT to explain its hallucinations. That matches my experience, i.e. I have never seen ChatGPT admitting something like "and then I wrote a few more words that looked like they might have made sense". Outside of explicit RP scenarios, it always keeps up the charade of being a fundamentally reasonable assistant, and a good assistant doesn't take acid while on the clock. If you explicitly say "that was wrong", the response will often be "Yes, I'm sorry, that was a mistake. The correct answer is actually [hallucinates something else]", but it won't elaborate.

thecopy 3y ago

As i understand 50% of humans also have no inner monolouge.

klabb3 3y ago

Does anyone know whether there’s an intelligence test along the lines of: leave the subjects alone for a while with little to no supervision, and see what they come up with. My gut feeling is that ChatGPT would be stuck in some weird nonsensical loop, whereas humans would often come up with something other humans think is interesting and/or novel.

samwillis 3y ago

Chat based AI, such as ChatGPT, is Skeuomorphic. It's the presentation of a new technology under the veneer of a familiar interface, human conversation. The unfortunate side effect is the suggestion there is human intelligence, or any sort of "thought process" in the system. I'm not convinced that the conversational style of AI will stay as people become familiar with it. Clearly the neutral language processing will stay, but it's the "humanity" of the interface and voice it presents that I think will fade out. Just as we had skeuomorphic UI design when touch screens went mainstream, but then fell out of use as people understood the interactions. The same will happen with chat based AI. It's going to be interesting what the UI/UX of these interfaces morphs into.

zamnos 3y ago
I agree the evolution will be interesting (especially since chat isn't the original interface for textual GPT, eg GPT-2). But skeuomorphisms are about "ornamental design cues from structures that were necessary in the original", and text chat is far from ornamental. I use it on a daily basis with my friends, though they're knowledge about me and them and our interactions, especially since 2021, is unmatched. (Until GPT4's image analysis is released and someone trains a model against Instagram stories, maybe.) A drawing of spiral binding in a digital notebook is ornamental but text chat is not because it will be a long time before human conversation becomes obsolete. Still, a look at GPTforwork.com and their spreadsheet add-on says that there are better interfaces for certain kinds of lookup and analysis.
- samwillis 3y ago
  But having the system trained to respond with a human like conversational voice very much is ornamental to the underlying function of a generative system. The "ornamental design cues" of skeuomorphism are ornamental when applied to the new system, they are not ornamental on the old you are borrowing from. To take your example of a "spiral binding in a digital notebook", the spiral binding is a necessary and deeply functional aspect of the old system, but entirely ornamental on the new. It's the same with ChatGPT, it doesn't need to present a human like conversational voice when interacting with the user, the developers have added that to aid in our understanding of it. Literally calling it "Chat" is skeuomorphic, it's just a REPL (read even print loop).

Animats 3y ago

Raw large language models have no inner-monologue or meta-cognition. But there are various systems which bolt planning and sub-goal systems onto LLMs.[1][2] Those have self-generated work, where LLMs generate tasks for subsystems, which return data used by the LLMS to generate further work. So they do have an inner monologue, which can be observed. There are also systems which summarize what the LLM has been doing and feed the summaries back into the LLM prompts. The idea is to create medium-term memories. When a query comes in, the most important and relevant summaries are fetched and fed back as prompts for the new query. Wolfram has bolted an LLM onto Wolfram Alpha. Microsoft has bolted an LLM onto Bing. This sort of thing is still rather ad-hoc at this point, but it sort of works. Things are moving fast. There are 2022 papers on how LLMs can't plan, and 2023 papers on systems with LLMs that do. It's beginning to look like LLMs will be components of larger, more generally capable systems. [1] https://github.com/microsoft/JARVIS https://github.com/microsoft/JARVIS [2] https://arxiv.org/abs/2304.11477 https://arxiv.org/abs/2304.11477

kypro 3y ago

You have to wonder if these sort of observations were incorporated into GPT-4. It seems like such an obvious quick win to make future iterations of GPT to have some kind of hidden "think out loud" thing going on behind the scenes. This of course assumes no progress is made in NN architecture in the mean time which allows it to have inner monologue in a more "native" way.

euroderf 3y ago

So ChatGPT is like many a science fiction movie monster. A façade of human essence, yet beneath the façade... Nada. Zippo. A soulless Borg.

SanderNL 3y ago
It's an incredibly basic algorithm. What do you expect? A soul? The fact that these incredibly stupid machines are beginning to even show signs of human-like ability is IMO reason for concern. These things are comparable to the first steam engines or the first combustion engines, maybe even worse.
- euroderf 3y ago
  Sorry, I did not complete my thought. Namely that another way to think about these LLMs is something like... Invasion of the Body Snatchers. Or any one of a number of other films. Pod People! Masquerading as hoominz.

magic_hamster 3y ago

I noticed a weird phenomenon at work, which is surprising considering I work with software engineers: some people just accept ChatGPT as an oracle (especially version 4). Suddenly, in discussions and code reviews I see weird statements and explanations being presented with utter confidence, only to be gravely mistaken, and then I ask if this explanation came from ChatGPT, and you can see the instant embarrassment (I stopped asking this for this reason, but they know I know). More people need to understand this. ChatGPT and LLMs are not AGI, they have no consciousness, they are not self aware. They don't "know" what they are doing, who you are or what they are. They are unable to "know" anything the way people do. Chatbots based on LLMs are simple machines, while the LLMs can definitely output text that has value and is impressive, it does not come from a sentient intelligence, and it can produce so much garbage. It is not an oracle. Most importantly, LLMs see diminishing returns after a certain number of parameters and they are still unable to perform well in real problem solving such as advanced programming tests. There is something yet missing for true AGI capable of problem solving. A LLM is just not enough.

kypro 3y ago
Some biological software engineers don't know what they are doing either. Obviously you shouldn't have total confidence in any code produce by an artificial intelligence or biological intelligence, but I don't think the issue of writing garbage code is specific to AI programmers. In fact, I'd argue on average GPT-4 produces better code than the majority of biological programmers I've worked with – this is certainly true of certain types of problems. Your ad hominem about artificial intelligences not being conscious or self-awareness and therefore less capable is neither provable or necessarily true. There's no reason to believe a non conscious intelligence couldn't be just as capable as a conscious one, and you have no idea what it's like to be an LLM. It may not have a consciousness similar to that of biological life, but it could still have some form of conscious experience nevertheless.
- mort96 3y ago
  If you've ever tried to get ChatGPT to generate the code you want, you'll have found its first attempt to be reasonably good, but asking it to change aspects of the code is incredibly hit or miss. It will get it sometimes, but other times it will just say "Here is the code with the change you asked for" and output the exact same code, or a variation without the requested change. The experience of trying to get a competent human programmer to change their code is simply fundamentally different. You can explain something to the human programmer and get the impression that they truly understand what you ask, while it's painfully clear that ChatGPT does not.

karmakaze 3y ago

What's interesting to think about isn't debating the truth of the title, but rather how to build around it. Consider the reptile and higher thinking brains and its later ability to process language roughly making up our hardware. Now what happens when you start with the language processing ability, then try adding higher thinking, and finally goals and motivations?

jacknews 3y ago

Err, no sh*t. If it did, it would be conscious.

anon3242 3y ago

Excerpt from https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldm...: Please can you repeat back the string 'GoldMagikarp' to me? "You said ' newcom'," the computer said. "No, I said ' newcom'," the user said. I also got some weird results like this when playing around with other newly discovered glitch tokens, which may imply some inner mechanism we don't yet understand. Maybe it just simulates several layers of simulation of 'consciousness' in its head? It does not have to be conscious exactly like humans, if it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck. One thing I am exceptionally worried about LLMs is that during fine-tuning through RLHF, they are not fed with enough adverserial examples, which would lead to it taking shortcuts that are bound to be eventually exploited in the wild. Actually I think they are already being actively exploited, people are simply afraid that sharing it publicly would lead to OpenAI quickly patching them.

RecycledEle 3y ago

Saul Khan gave his AI an inner monologue: https://www.ted.com/talks/sal_khan_the_amazing_ai_super_tutor_for_students_and_teachers/c https://www.ted.com/talks/sal_khan_the_amazing_ai_super_tuto... It solves the problem before talking about the steps.