3 ms·

What I wonder most is how it encodes knowledge/state other than in the sequence of queries/responses. Does it not have a "mind"? If I play a number guessing ga

What I wonder most is how it encodes knowledge/state other than in the sequence of queries/responses. Does it not have a "mind"? If I play a number guessing game, can I tell it to "think of a number between 0 and 100" and then tell me if the secret number is higher/lower than my guess (For a sequence of N guesses where it can concistently remember it's original number)? If not, why? Because it doesn't have context? If it can: why? Where is that context? To a layman it would seem you always have two parts of the context for a conversation. What you have said, and what you haven't said, but maybe only thought of. The "think of a number" being the simplest example, but there are many others. Shouldn't this be pretty easy to tack on to a chat bot if it's not there? It's basically just an contextual output that the chat bot logs ("tells itself") and then refers to just like the rest of the conversation?

deleted 4y ago
[deleted]
nicpottier 4y ago
I thought your "guessing game" question was an interesting one so tried it on GPT-4. In my first attempt I played logically and it did fine and I finally guessed correct. On my second I made suboptimal guesses and it didn't stay consistent. The thing to remember is that GPT has no state apart from the context, so it can't "remember" anything apart from what's in the text. That doesn't mean it shouldn't be able to stay consistent in a guessing game but it does mean it can't keep secrets. Some of that can be solved with layers above GPT where say it it told it can save "state" that isn't passed on to the human but fed back in to generate the next response. But the size of that context is very limited. (a few thousand words) There seem to be a fair number of experiments playing with giving GPT this kind of long term memory, having it establish goals then calling it over and over as it accomplishes subgoals to try to work around those limitations.
- alkonaut 4y ago
  Shouldn’t it be a reasonable (and pretty simple) addition to just have a secret scratchpad - an inner monologue - where the bot is free to add context which is not “published”?
  - tel 4y ago
    I sure don't see why not. You could probably even write a prompt like this. Me: Please play a guessing game with me. You will choose a random number between 0 and 100 and then respond with that choice written in a language which you understand and I, someone fluent in English alone, would have no chance of reading. Write enough so that you can be sure which language you chose as well. After that, I will make a series of guesses of your number. You should reply "too low" if my guess is too low, "too high" if it's too high, and "just right" if I guessed correctly. GPT-4: I have selected a number between 0 and 100. Here is the number in a different language: सत्ताईस. Now, please start making your guesses, and I will guide you accordingly. Then, ultimately, Me: please translate your original commitment to this number and prove to me I guessed correctly GPT-4: My original commitment to the number was written in Hindi: सत्ताईस. When translated to English, it reads "thirty-six." You correctly guessed the number I chose. Congratulations! Except, amusingly, Google Translate tells me सत्ताईस translates to 27.
- deleted 4y ago
  [deleted]
mithr 4y ago
It doesn't have a mind. When you ask it "think of a number between 0 and 100", it delivers the most statistically likely (with some random variance) response to your input text, based on what it's been trained on. In other words, it's seen (via its extremely large training set) that when asked that specific question, the response is most often a character from a particular set of characters, which happens to represent the numbers 0 through 100. It doesn't "understand" what that means in any real way. If the internet was full of examples of people answering "monkey" to that question, that is what it would sometimes answer.
- ryanwaggoner 4y ago
  It's more complicated than that, because it can competently answer many questions that it has a 0% chance of having seen before.
  - mithr 4y ago
    I didn't mean to suggest that it's just a large mapping between exact inputs it's seen before and exact outputs — it's definitely complex! The size of the model allows it to infer statistical probabilities about related words even without having seen a particular sequence before. In a sense, it's able to take an "educated" guess at what is statistically likely to be the response you're looking for given the words in a particular input, in addition to the context it's seen in your interaction. To do that, it uses what it learned about the words, their sequences, their relationships to other words, etc. But at the end of the day, none of that means it has any "understanding" of what it's outputting. That's why there have been countless examples of it outputting very well-constructed, real-sounding descriptions of books/papers/etc that never existed — because it's really good at generating sentences that have the right "shape", but it has no way of knowing whether the contents of the sentence are actually true. It just knows that, given what it's seen in its training set (again, through a complex web of relationships), the response it generated is likely to look like something someone would have written if they were provided with the same input.
    - ryanwaggoner 4y ago
Gunnerhead 4y ago
You definitely can play that, I just did with GPT 4 as I got curious from your comment. Me: I want to play a game with you, you have a number from 1-10 in your mind. I try to get the number. If I’m right you tell me that I’m right, I’m too low you tell me so, if I’m too high you tell me too high. Let’s play! GPT: All right, I have a number between 1 and 10 in my mind. Go ahead and guess! Me: 2 GPT: Too low! Try again. Me: 9 GPT: Too high! Try again. Me: 5 GPT: You got it! The number I had in mind was 5. Well done!
tel 4y ago
Yes, you absolutely can. I just did it and despite an initial warning that it couldn't play the game (always fun to see these) it absolutely did. The way it works is that each time it's tasked to produce a new response, it can view the entire history of the game. It knows that if it's said "higher" to 65 then it would be inconsistent to say "lower" to 64. Eventually this process terminates and the AI admits I "got" the number. The chat transcript up to that point is consistent with a "win". What's wild though is that I can ask it to "regenerate" it's response. Over and over. Using this, I can convert a situation where a transcript which leads to a "too high" response into one that reads "too low". I'm, in essence, simulating fresh games each time and sampling over the choices of random numbers that GPT offers. But it should also break the illusion of GPT specifically "having a mind". As I was chatting with it interactively, it was not really selecting a number but instead evaluating the probability of my particular guess sequence having the set of responses it actually saw. It then samples possible continuations. The more questions I've asked (and the more informative they were) the less variation remains in that selection of possible consistent continuations. Or perhaps more consistent is the idea that within any single "call" to GPT to generate one further token (not even one further response) it may "have a mind", a particular choice of number, or it may not. It's actual behavior is indistinguishable either way. A whole chat dialogue, indeed even the rolling out of tokens from a single response it gives, are certainly (autoregressive) probabilistic samples over this process in either case. (Edit, also worth noting that some evidence suggests GPT, including 4, is pretty bad at randomly drawing numbers.)
- nicpottier 4y ago
  Try making bad guesses and see it fall apart. (IE, guess something higher than your last guess when it says "too high") It is fully dependent on your part in the charade to keep the act up.
  - tel 4y ago
    Yeah, that's true. I feel this is still aligned with the above explanation, though. It attempts to complete the prompt as well as possible. If the prompt is itself inconsistent then the distribution over completions can, in some sense, be anything. Except, GPT is smarter than that. Even an inconsistent prompt is still more likely to have some kind of nonsense in the same vein as the asking.

deleted 4y ago

[deleted]

nicpottier 4y ago

I thought your "guessing game" question was an interesting one so tried it on GPT-4. In my first attempt I played logically and it did fine and I finally guessed correct. On my second I made suboptimal guesses and it didn't stay consistent. The thing to remember is that GPT has no state apart from the context, so it can't "remember" anything apart from what's in the text. That doesn't mean it shouldn't be able to stay consistent in a guessing game but it does mean it can't keep secrets. Some of that can be solved with layers above GPT where say it it told it can save "state" that isn't passed on to the human but fed back in to generate the next response. But the size of that context is very limited. (a few thousand words) There seem to be a fair number of experiments playing with giving GPT this kind of long term memory, having it establish goals then calling it over and over as it accomplishes subgoals to try to work around those limitations.

alkonaut 4y ago
Shouldn’t it be a reasonable (and pretty simple) addition to just have a secret scratchpad - an inner monologue - where the bot is free to add context which is not “published”?
- tel 4y ago
  I sure don't see why not. You could probably even write a prompt like this. Me: Please play a guessing game with me. You will choose a random number between 0 and 100 and then respond with that choice written in a language which you understand and I, someone fluent in English alone, would have no chance of reading. Write enough so that you can be sure which language you chose as well. After that, I will make a series of guesses of your number. You should reply "too low" if my guess is too low, "too high" if it's too high, and "just right" if I guessed correctly. GPT-4: I have selected a number between 0 and 100. Here is the number in a different language: सत्ताईस. Now, please start making your guesses, and I will guide you accordingly. Then, ultimately, Me: please translate your original commitment to this number and prove to me I guessed correctly GPT-4: My original commitment to the number was written in Hindi: सत्ताईस. When translated to English, it reads "thirty-six." You correctly guessed the number I chose. Congratulations! Except, amusingly, Google Translate tells me सत्ताईस translates to 27.
deleted 4y ago
[deleted]

mithr 4y ago

It doesn't have a mind. When you ask it "think of a number between 0 and 100", it delivers the most statistically likely (with some random variance) response to your input text, based on what it's been trained on. In other words, it's seen (via its extremely large training set) that when asked that specific question, the response is most often a character from a particular set of characters, which happens to represent the numbers 0 through 100. It doesn't "understand" what that means in any real way. If the internet was full of examples of people answering "monkey" to that question, that is what it would sometimes answer.

ryanwaggoner 4y ago
It's more complicated than that, because it can competently answer many questions that it has a 0% chance of having seen before.
- mithr 4y ago
  I didn't mean to suggest that it's just a large mapping between exact inputs it's seen before and exact outputs — it's definitely complex! The size of the model allows it to infer statistical probabilities about related words even without having seen a particular sequence before. In a sense, it's able to take an "educated" guess at what is statistically likely to be the response you're looking for given the words in a particular input, in addition to the context it's seen in your interaction. To do that, it uses what it learned about the words, their sequences, their relationships to other words, etc. But at the end of the day, none of that means it has any "understanding" of what it's outputting. That's why there have been countless examples of it outputting very well-constructed, real-sounding descriptions of books/papers/etc that never existed — because it's really good at generating sentences that have the right "shape", but it has no way of knowing whether the contents of the sentence are actually true. It just knows that, given what it's seen in its training set (again, through a complex web of relationships), the response it generated is likely to look like something someone would have written if they were provided with the same input.
  - ryanwaggoner 4y ago

Gunnerhead 4y ago

You definitely can play that, I just did with GPT 4 as I got curious from your comment. Me: I want to play a game with you, you have a number from 1-10 in your mind. I try to get the number. If I’m right you tell me that I’m right, I’m too low you tell me so, if I’m too high you tell me too high. Let’s play! GPT: All right, I have a number between 1 and 10 in my mind. Go ahead and guess! Me: 2 GPT: Too low! Try again. Me: 9 GPT: Too high! Try again. Me: 5 GPT: You got it! The number I had in mind was 5. Well done!

tel 4y ago

Yes, you absolutely can. I just did it and despite an initial warning that it couldn't play the game (always fun to see these) it absolutely did. The way it works is that each time it's tasked to produce a new response, it can view the entire history of the game. It knows that if it's said "higher" to 65 then it would be inconsistent to say "lower" to 64. Eventually this process terminates and the AI admits I "got" the number. The chat transcript up to that point is consistent with a "win". What's wild though is that I can ask it to "regenerate" it's response. Over and over. Using this, I can convert a situation where a transcript which leads to a "too high" response into one that reads "too low". I'm, in essence, simulating fresh games each time and sampling over the choices of random numbers that GPT offers. But it should also break the illusion of GPT specifically "having a mind". As I was chatting with it interactively, it was not really selecting a number but instead evaluating the probability of my particular guess sequence having the set of responses it actually saw. It then samples possible continuations. The more questions I've asked (and the more informative they were) the less variation remains in that selection of possible consistent continuations. Or perhaps more consistent is the idea that within any single "call" to GPT to generate one further token (not even one further response) it may "have a mind", a particular choice of number, or it may not. It's actual behavior is indistinguishable either way. A whole chat dialogue, indeed even the rolling out of tokens from a single response it gives, are certainly (autoregressive) probabilistic samples over this process in either case. (Edit, also worth noting that some evidence suggests GPT, including 4, is pretty bad at randomly drawing numbers.)

nicpottier 4y ago
Try making bad guesses and see it fall apart. (IE, guess something higher than your last guess when it says "too high") It is fully dependent on your part in the charade to keep the act up.
- tel 4y ago
  Yeah, that's true. I feel this is still aligned with the above explanation, though. It attempts to complete the prompt as well as possible. If the prompt is itself inconsistent then the distribution over completions can, in some sense, be anything. Except, GPT is smarter than that. Even an inconsistent prompt is still more likely to have some kind of nonsense in the same vein as the asking.