3 ms·

What's to stop a Markov chain or very simple language model (something that prompting doesn't work on, like an RNN with a modest hidden state) from meeting this

What's to stop a Markov chain or very simple language model (something that prompting doesn't work on, like an RNN with a modest hidden state) from meeting this definition? Because I wouldn't buy that either of these are aware, and both of them predict the next token.

TeMPOraL 4y ago
I think it's a matter of degree, not of kind. That is, the complexity of the model predicting the next token matters, and so does the space of possible tokens. Ultimately, I think the definite answer would require solving the halting problem. The approximate answer we should be able to get when we let models interact with some complex environment over time. That is, the model predicts the next token, which may cause some change of a shared state, and then the model receives a response.
- maxbond 4y ago
  I dunno. Is there a testable prediction of this hypothesis? I just don't really see why I should believe it other than that chatting to LLMs is compelling, which isn't enough for me. This seems like an answer to the question, "why is the LLM so compelling?" And not "what makes something conscious/aware?" Which is an important question but feels like not quite the right one to ask and then come to this conclusion. And it bothers me if I try to accept an idea where I don't know what conditions I'd reject it (eg, I don't know what evidence would cause me to reject this hypothesis, I can't see a test I can perform to next reject it).
grumbel 4y ago
Simple models will very quickly produce content that doesn't fit. Just stringing a bit of grammar and a dictionary together will produce locally correct sentences, but it will completely fail at generate larger texts with context. GPT in contrast knows what it was talking about five sentences ago and can refer to it in newly formed sentences.
jodrellblank 4y ago
Go and write a Markov chain, you need some text, a hashtable and a few minutes, you'll quickly see that they generate rubbish which doesn't fit anywhere, no sense of context, no sense of 'chat', no sense of intelligence to them. ChatGPT 'predicts the next token' so much better than other 'next token predictors' that using that as the comparison for a low effort dismissal is like saying "what's to stop a human tracking a criminal by scent, humans have noses and dogs have noses so they must be the same".
- maxbond 4y ago
  I'm aware of how these models work (I have trained simple LSTM text generators); I didn't dismiss anything (other than that simple models aren't aware, which you don't seem to contest). If we agree that Markov chains aren't aware, then surely we also agree this definition of what makes them aware is either incomplete or incorrect. Predicting the next token turns out to be quite profound, I'm just not convinced, at this time, it's sufficient to be aware of conscious. If you have a reason I should believe it, I'm all ears.
  - grumbel 4y ago
    Predicting the next token perfectly would make a chatbot an omniscient god, as it could tell you things like the lottery numbers from next week. Chatbots "just" being a token predictors does not limit their power or intelligence in any way. It however also doesn't tell you how good they are at their job. If you want to test how good a chatbot is at predicting tokens, well, you just have to test it. You can throw a Winograd schema challenge at it, or let it take part in code competition or exams, let it solve puzzles and all that. Quick simple test will for example show you that ChatGPT isn't any good at calculating, as the answers will just be wrong. As for awareness, we know what ChatGPT is aware of: Its 4000 token history and whatever information it learned from the training data. That's the grand total of its existence, it doesn't get any other input, it doesn't perceive time, it doesn't learn, that history is all there is to it. And we can call that awareness as ChatGPT is able to perform correct prediction based on that token history. For consciousness there are still a lot of basic parts missing, such as the perception of time, senses to act with some external world, the ability to think for itself without getting explicitly prompted and all that.
rrobukef 4y ago
Remember the sensation of ELIZA? People anthropomorphised responses following a set of understandable rules. I wouldn't buy it, but you can sell it to plenty of people.

TeMPOraL 4y ago

I think it's a matter of degree, not of kind. That is, the complexity of the model predicting the next token matters, and so does the space of possible tokens. Ultimately, I think the definite answer would require solving the halting problem. The approximate answer we should be able to get when we let models interact with some complex environment over time. That is, the model predicts the next token, which may cause some change of a shared state, and then the model receives a response.

maxbond 4y ago
I dunno. Is there a testable prediction of this hypothesis? I just don't really see why I should believe it other than that chatting to LLMs is compelling, which isn't enough for me. This seems like an answer to the question, "why is the LLM so compelling?" And not "what makes something conscious/aware?" Which is an important question but feels like not quite the right one to ask and then come to this conclusion. And it bothers me if I try to accept an idea where I don't know what conditions I'd reject it (eg, I don't know what evidence would cause me to reject this hypothesis, I can't see a test I can perform to next reject it).

grumbel 4y ago

Simple models will very quickly produce content that doesn't fit. Just stringing a bit of grammar and a dictionary together will produce locally correct sentences, but it will completely fail at generate larger texts with context. GPT in contrast knows what it was talking about five sentences ago and can refer to it in newly formed sentences.

jodrellblank 4y ago

Go and write a Markov chain, you need some text, a hashtable and a few minutes, you'll quickly see that they generate rubbish which doesn't fit anywhere, no sense of context, no sense of 'chat', no sense of intelligence to them. ChatGPT 'predicts the next token' so much better than other 'next token predictors' that using that as the comparison for a low effort dismissal is like saying "what's to stop a human tracking a criminal by scent, humans have noses and dogs have noses so they must be the same".

maxbond 4y ago
I'm aware of how these models work (I have trained simple LSTM text generators); I didn't dismiss anything (other than that simple models aren't aware, which you don't seem to contest). If we agree that Markov chains aren't aware, then surely we also agree this definition of what makes them aware is either incomplete or incorrect. Predicting the next token turns out to be quite profound, I'm just not convinced, at this time, it's sufficient to be aware of conscious. If you have a reason I should believe it, I'm all ears.
- grumbel 4y ago
  Predicting the next token perfectly would make a chatbot an omniscient god, as it could tell you things like the lottery numbers from next week. Chatbots "just" being a token predictors does not limit their power or intelligence in any way. It however also doesn't tell you how good they are at their job. If you want to test how good a chatbot is at predicting tokens, well, you just have to test it. You can throw a Winograd schema challenge at it, or let it take part in code competition or exams, let it solve puzzles and all that. Quick simple test will for example show you that ChatGPT isn't any good at calculating, as the answers will just be wrong. As for awareness, we know what ChatGPT is aware of: Its 4000 token history and whatever information it learned from the training data. That's the grand total of its existence, it doesn't get any other input, it doesn't perceive time, it doesn't learn, that history is all there is to it. And we can call that awareness as ChatGPT is able to perform correct prediction based on that token history. For consciousness there are still a lot of basic parts missing, such as the perception of time, senses to act with some external world, the ability to think for itself without getting explicitly prompted and all that.

rrobukef 4y ago

Remember the sensation of ELIZA? People anthropomorphised responses following a set of understandable rules. I wouldn't buy it, but you can sell it to plenty of people.