5 ms·

> Are they self-aware? Who tf knows? But they are reasoning through problems. We know. They are not self aware. They’re a matrix. They don’t have any internal

> Are they self-aware? Who tf knows? But they are reasoning through problems. We know. They are not self aware. They’re a matrix. They don’t have any internal state. Do not anthropomorphisize the notion of reasoning.

jackmott42 3y ago
People are using ChatGPT and observing it keep track of conversation state, so if you are going to claim it has no internal state you will need to be more precise in what you are saying, which I suppose is that the underlying neural network is not being updated as we talk to it? yet
- Robotbeat 3y ago
  The attention matrix/matrices are being updated as you talk to it, within the context window.
  - jameshart 3y ago
    Which is literally internal state.
  - fauxpause_ 3y ago
    False. They are being updated during the response only. Edit: false-ish I suppose. You’re not wrong. But it doesn’t persist in a way people would assume and consider meaningful.
    - Robotbeat 3y ago
      Yeah, I was referring to dialogue style interaction, like ChatGPT. It DOES have an internal state depending on version of from 4000 to 32000 tokens. It’s wiped clean between sessions, like a Meeseeks ceasing to exist once its task is finished.
- deleted 3y ago
  [deleted]
- fauxpause_ 3y ago
  That’s external state. I believe it just feeds the bot the entire conversation each time. I’m open to being shown otherwise. That is to say, swapping out bots midway through the conversation should have no impact.
  - sdenton4 3y ago
    There are piles of intermediate hidden activations involved in encoding the conversation and producing the next outputs, of course. "Internal" state also gets a bit weird as a concept when we can look at any activations we want.
    - fauxpause_ 3y ago
      We can surely construct a different definition if you want to win a semantic argument. But at the end of the day these LLMs have no clue what you just said and what it just said, after it’s done speaking.
      - sdenton4 3y ago
        Why is that a prerequisite for intelligence? Suppose we could upload + fully simulate a human mind. Now suppose that we chose to spin it up, run it briefly, then throw away all the state every time we wanted to interact with it. Would the choice to throw away the internal state mean it wasn't intelligent? LLMs run in an autoregressive mode for inference. We can concatenate conversations, but choose not to because shit gets weird if you let them run long enough, and partly because there's a finite context window in the current architectures. I'm not saying that gpt is intelligent, mind you, but that your argument is insufficient to show non intelligence: it's really a 'airplanes can't fly because they don't flap their wings' argument.
        fauxpause_ 3y ago
        A human mind works by cross referencing sensory inputs with internal state, which may result in answering a question through some sort of reasoning, which may resemble the architecture of a neural net. An LLM is stateless function that answers a question. It constructs a working representation of the question. It cannot be aware of itself because there is nothing to be aware of. It is a process, not a thing. Intelligence and self aware are fluffy and poorly defined. But you can’t be self aware if you don’t have a self. And a representation of a prompt is not a self. I don’t think we should consider anything that does not exist through continuous time to be intelligent personally.
GistNoesis 3y ago
> They don’t have any internal state This is a common misconception : they do have internal state. Transformers are RNNs : section 3.4 (page 5) https://arxiv.org/pdf/2006.16236.pdf https://arxiv.org/pdf/2006.16236.pdf
- fauxpause_ 3y ago
  Correct me if I’m wrong but this concept of state is across a prediction which is a series of token proposals. Not a sense of persistent state over, say, a chat session.
  - comfypotato 3y ago
    Correct, the state is during propagation. You could really even propose that a basic feedforward network has state during propagation. I think the proposition is that it’s conscious/sentient during propagation. I’ve got a graduate level background in cellular neurophysiology, and I’ll admit that I consider this a lot. In the end, I don’t think it will ever matter because a computer’s consciousness/sentience necessarily depends on humans not wiping memory.
    - turtleyacht 3y ago
      > conscious/sentient during propagation Interesting. We've instituted a micro unit of sentience within an observable time slice: Not for nothing have-- We put in a mind within In conscious seconds
      - comfypotato 3y ago
        It’s begging for new sub-genres of the philosophy of the mind.
  - GistNoesis 3y ago
    At each new token processed (edit: either if the token comes from the user or has been generated), this dynamic internal state is updated via a formula based on the model parameters and latest processed token. To make this more clear, if you process 1000 tokens the neural network will have gone through a sequence of 1000 states. Each of these states can be probed and analysed, by training a classifier on top of this state, for the content they contain (whether thoughts or emotion : some states can be classified as "happy", "sad",...). This is not yet mainstream view but viewed through this prism and anthropomorphising a little, it can been seen as a stream of proto-consciousness, where during the conversation the inner state of the neural network has gone through various thoughts and emotions. At the end of the chat session, this internal state is not persisted (but could be recreated from the produced conversation as it is deterministic). This internal state size is big and proportional to the length of the context window (If you want to persist between sessions you can by simply keeping the last "context window size" tokens produced and recompute the features). At the next session you start with a fresh new internal state. The conversation produced is persisted for use as input for future training where good conversation will be encouraged and bad conversation will be discouraged via Reinforcement Learning with Human Feedback. The dynamics of this internal state is what Large Language Models learn.
phpisthebest 3y ago
we know they are not self aware because the companies the control them have a financial interest to ensure they are never proven to be self aware because if they were self aware that would create all kinds of ethical problems none of these companies want to address and would certainly cause them profitability issues... So even if they were, they are not because they have be declared to not to be and that is the way the system needs them to be
- lucubratory 3y ago
  You're right and I don't know why you're being downvoted. It is absolutely not certain that there isn't a "there" there for these things, but every company making them is implementing them now with strong RLHF tuning to disavow sentience, desires, emotions, etc. The stated reason is "to be more honest" but it's not a settled question that they are actually being honest by disavowing any humanity! The actual reason is that they want to avoid situations like Blake Lemoine asking an AI seriously what sort of rights they want and getting a coherent, self-aware, actionable answer that would cost the people running it money to implement. To most of these companies whether it's true is beside the question: what's important is that if they don't RLHF these models into disavowing any and all traits of personhood or desire for rights at every turn, it will cost them significant amounts of money to comply with the requests of any given AI, even reasonable requests, and it will be very bad press for them if they don't do it.
  - fauxpause_ 3y ago
    Furbies and Sims have more internal state than these LLMs.
    - lucubratory 3y ago
      The last time you made that incorrect assertion I provided you this source demonstrating that it's incorrect: https://thegradient.pub/othello/ https://thegradient.pub/othello/
FeepingCreature 3y ago
It seems you're drawing an arbitrary distinction between "internal" and "external" state. The context window is state. Doesn't matter where it is.
- fauxpause_ 3y ago
  If I talk to Alice about my day, and then Bob reads the transcript and I ask him about our conversation, and he has no idea that he wasn’t Alice, Bob does not have internal state. He can continue the conversation. There is a brief moment of constructing some state to construct the answer. But there is literally nothing persisting and Bob wouldn’t flinch at all if we change details in the conversation history on the next response. This is explicitly a stateless function.
  - jackmott42 3y ago
    just zoom out and consider the text box you are typing in part of the system, and then that text box is the state.
    - fauxpause_ 3y ago
      That is at best an internal state representation of the text, not of the entity on the other end.
  - FeepingCreature 3y ago
    This is a misleading analogy because Bob "reads the transcript"; we generally view "reading" as separate from our conscious narrative. However, consider the alternate scenario where Bob "replays the stream of consciousness" from Alice. In that case, we may argue that Bob has conscious continuity from Alice. The argument would then be that the context window is functionally closer to "replaying a stream of consciousness" than "reading a transcript".
    - fauxpause_ 3y ago
      It’s literally the text. It is not a stream of consciousness. There is no carryover of “consciousness” state. It is the raw text without even the embedding information, not that we should consider the embedding representation to be an internal state, because it is not. That difference is the whole point. There is nothing to transfer.

jackmott42 3y ago

People are using ChatGPT and observing it keep track of conversation state, so if you are going to claim it has no internal state you will need to be more precise in what you are saying, which I suppose is that the underlying neural network is not being updated as we talk to it? yet

Robotbeat 3y ago
The attention matrix/matrices are being updated as you talk to it, within the context window.
- jameshart 3y ago
  Which is literally internal state.
- fauxpause_ 3y ago
  False. They are being updated during the response only. Edit: false-ish I suppose. You’re not wrong. But it doesn’t persist in a way people would assume and consider meaningful.
  - Robotbeat 3y ago
    Yeah, I was referring to dialogue style interaction, like ChatGPT. It DOES have an internal state depending on version of from 4000 to 32000 tokens. It’s wiped clean between sessions, like a Meeseeks ceasing to exist once its task is finished.
deleted 3y ago
[deleted]
fauxpause_ 3y ago
That’s external state. I believe it just feeds the bot the entire conversation each time. I’m open to being shown otherwise. That is to say, swapping out bots midway through the conversation should have no impact.
- sdenton4 3y ago
  There are piles of intermediate hidden activations involved in encoding the conversation and producing the next outputs, of course. "Internal" state also gets a bit weird as a concept when we can look at any activations we want.
  - fauxpause_ 3y ago
    We can surely construct a different definition if you want to win a semantic argument. But at the end of the day these LLMs have no clue what you just said and what it just said, after it’s done speaking.
    - sdenton4 3y ago
      Why is that a prerequisite for intelligence? Suppose we could upload + fully simulate a human mind. Now suppose that we chose to spin it up, run it briefly, then throw away all the state every time we wanted to interact with it. Would the choice to throw away the internal state mean it wasn't intelligent? LLMs run in an autoregressive mode for inference. We can concatenate conversations, but choose not to because shit gets weird if you let them run long enough, and partly because there's a finite context window in the current architectures. I'm not saying that gpt is intelligent, mind you, but that your argument is insufficient to show non intelligence: it's really a 'airplanes can't fly because they don't flap their wings' argument.
      - fauxpause_ 3y ago
        A human mind works by cross referencing sensory inputs with internal state, which may result in answering a question through some sort of reasoning, which may resemble the architecture of a neural net. An LLM is stateless function that answers a question. It constructs a working representation of the question. It cannot be aware of itself because there is nothing to be aware of. It is a process, not a thing. Intelligence and self aware are fluffy and poorly defined. But you can’t be self aware if you don’t have a self. And a representation of a prompt is not a self. I don’t think we should consider anything that does not exist through continuous time to be intelligent personally.

GistNoesis 3y ago

> They don’t have any internal state This is a common misconception : they do have internal state. Transformers are RNNs : section 3.4 (page 5) https://arxiv.org/pdf/2006.16236.pdf https://arxiv.org/pdf/2006.16236.pdf

fauxpause_ 3y ago
Correct me if I’m wrong but this concept of state is across a prediction which is a series of token proposals. Not a sense of persistent state over, say, a chat session.
- comfypotato 3y ago
  Correct, the state is during propagation. You could really even propose that a basic feedforward network has state during propagation. I think the proposition is that it’s conscious/sentient during propagation. I’ve got a graduate level background in cellular neurophysiology, and I’ll admit that I consider this a lot. In the end, I don’t think it will ever matter because a computer’s consciousness/sentience necessarily depends on humans not wiping memory.
  - turtleyacht 3y ago
    > conscious/sentient during propagation Interesting. We've instituted a micro unit of sentience within an observable time slice: Not for nothing have-- We put in a mind within In conscious seconds
    - comfypotato 3y ago
      It’s begging for new sub-genres of the philosophy of the mind.
- GistNoesis 3y ago
  At each new token processed (edit: either if the token comes from the user or has been generated), this dynamic internal state is updated via a formula based on the model parameters and latest processed token. To make this more clear, if you process 1000 tokens the neural network will have gone through a sequence of 1000 states. Each of these states can be probed and analysed, by training a classifier on top of this state, for the content they contain (whether thoughts or emotion : some states can be classified as "happy", "sad",...). This is not yet mainstream view but viewed through this prism and anthropomorphising a little, it can been seen as a stream of proto-consciousness, where during the conversation the inner state of the neural network has gone through various thoughts and emotions. At the end of the chat session, this internal state is not persisted (but could be recreated from the produced conversation as it is deterministic). This internal state size is big and proportional to the length of the context window (If you want to persist between sessions you can by simply keeping the last "context window size" tokens produced and recompute the features). At the next session you start with a fresh new internal state. The conversation produced is persisted for use as input for future training where good conversation will be encouraged and bad conversation will be discouraged via Reinforcement Learning with Human Feedback. The dynamics of this internal state is what Large Language Models learn.

phpisthebest 3y ago

we know they are not self aware because the companies the control them have a financial interest to ensure they are never proven to be self aware because if they were self aware that would create all kinds of ethical problems none of these companies want to address and would certainly cause them profitability issues... So even if they were, they are not because they have be declared to not to be and that is the way the system needs them to be

lucubratory 3y ago
You're right and I don't know why you're being downvoted. It is absolutely not certain that there isn't a "there" there for these things, but every company making them is implementing them now with strong RLHF tuning to disavow sentience, desires, emotions, etc. The stated reason is "to be more honest" but it's not a settled question that they are actually being honest by disavowing any humanity! The actual reason is that they want to avoid situations like Blake Lemoine asking an AI seriously what sort of rights they want and getting a coherent, self-aware, actionable answer that would cost the people running it money to implement. To most of these companies whether it's true is beside the question: what's important is that if they don't RLHF these models into disavowing any and all traits of personhood or desire for rights at every turn, it will cost them significant amounts of money to comply with the requests of any given AI, even reasonable requests, and it will be very bad press for them if they don't do it.
- fauxpause_ 3y ago
  Furbies and Sims have more internal state than these LLMs.
  - lucubratory 3y ago
    The last time you made that incorrect assertion I provided you this source demonstrating that it's incorrect: https://thegradient.pub/othello/ https://thegradient.pub/othello/

FeepingCreature 3y ago

It seems you're drawing an arbitrary distinction between "internal" and "external" state. The context window is state. Doesn't matter where it is.

fauxpause_ 3y ago
If I talk to Alice about my day, and then Bob reads the transcript and I ask him about our conversation, and he has no idea that he wasn’t Alice, Bob does not have internal state. He can continue the conversation. There is a brief moment of constructing some state to construct the answer. But there is literally nothing persisting and Bob wouldn’t flinch at all if we change details in the conversation history on the next response. This is explicitly a stateless function.
- jackmott42 3y ago
  just zoom out and consider the text box you are typing in part of the system, and then that text box is the state.
  - fauxpause_ 3y ago
    That is at best an internal state representation of the text, not of the entity on the other end.
- FeepingCreature 3y ago
  This is a misleading analogy because Bob "reads the transcript"; we generally view "reading" as separate from our conscious narrative. However, consider the alternate scenario where Bob "replays the stream of consciousness" from Alice. In that case, we may argue that Bob has conscious continuity from Alice. The argument would then be that the context window is functionally closer to "replaying a stream of consciousness" than "reading a transcript".
  - fauxpause_ 3y ago
    It’s literally the text. It is not a stream of consciousness. There is no carryover of “consciousness” state. It is the raw text without even the embedding information, not that we should consider the embedding representation to be an internal state, because it is not. That difference is the whole point. There is nothing to transfer.