3 ms·

A question to those who better understand the technology. My understanding is that the neural weightings in the LLM are established during training, but don't c

by simonh 4y ago

A question to those who better understand the technology. My understanding is that the neural weightings in the LLM are established during training, but don't change (or maybe don't materially change?) during the progress of a chat. At any given moment when generating a response the LLM uses the input of the session text, for that session only, right up to it's last response word, in order to predict what the next word should be. So basically the LLM neural net itself doesn't store any persistent state for the current chat. The only state specific to the current session is in the session text to that point. Is that correct?

MonkeyMalarky 4y ago
This is my understanding as well. The amount of mystical projection by users and journalists when talking about it has been astounding. I guess the same could be said of Bard and that one googler who lost their mind and got fired. I guess we'll have to teach future children to "not project real emotions on to the simula" alongside "just because you read it on the internet doesn't mean it's true".
- simonh 4y ago
  Well, I'm thinking I must be missing some nuance, it seems like there must be some session state stored in the LLM somehow, otherwise it would have to re-parse the entire chat session from scratch to generate each succeeding token. But maybe it does do that. I'm trying to understand if it is learning, or building a model of the world in any persistent way how that is happening and where it exists, if at all.
- hrunt 4y ago
  "Please don't anthropomorphize the AI. It doesn't like that."
  - MonkeyMalarky 4y ago
    Given how delusional people can be, I expect we'll have billboards from People for the Ethical Treatment of Artificial Intelligences (PETAi) with messages like "You wouldn't `systemctl halt` a human child..." by the year 2035.
    - bheadmaster 4y ago
      It's funny that you call them delusional, when we both know that the problem of AI consciousness is unsolved (and might even be unsolvable). I, for one, would most likely become a member of PETAi as soon as I was convinced, without reasonable doubt, that AI had true consciousness like me. I don't see any fundamental reason why matrix multiplication couldn't serve as a vessel to consciousness, but a bunch of chemical reactions in neurons could.
ibejoeb 4y ago
You're basically describing inference, which is using the model to get an output. You're correct that that has no impact on the parameters of the model. One way to arrive at a chat-like "chain of thought" experience is to iteratively build up a prompt based on the previous interactions and feed that in as a new, big prompt on each successive turn.
GistNoesis 4y ago
It's more subtle than that : weights don't change, but the internal state does change at each new word (which allows it to perform few shots learning such as taking and following instructions given in a prompt). This internal state is defined implicitly by the weights that don't change applied to the prompt and generated up to now text (which does change). This internal state is quite big ("dim of the features" times "number of layers" times "current length of text"), and represent an expended view of the relevant context of the conversation, that include things both low level features like what's the previous word, and higher level features like what's the tone and mood of the conversation and what's the direction this conversation is aiming at, so that you can predict what the next word should be accurately. This allow to have "chain of thought" reasoning. This sequence of internal states maybe can be seen as a form of proto stream of consciousness, but this is a controversial opinion to hold. Everything usually is not persisted, except for the conversation themselves that will almost certainly be used in subsequent training datasets where they will be used to improve further models. To keep anthropomorphising, it's like you have plenty of independent chat sessions everyday where you are free to bounce thoughts as they spring into your mind. During the night, some external entity (which are often other models that have been trained to distinguish good from bad) evaluate which conversations were good and which conversations were bad, and make you dream and replay the conversation you had during the day and to updates your world model weights. And the next day you woke up with a tendency to produce better conversations, because of the conversation of yesterday.
- simonh 4y ago
  Thank you very much, that's great. So there is a mutable working memory for the session that's persistent through the session. That's something that had been missing from previous explanations I'd seen.
  - GistNoesis 4y ago
    This mutable working memory is implicit. It comes from the transformer architecture. What the neural network weight are encoding is the dynamic of this implicit working memory such that it produces the desired output. If you use something like LSTM instead of transformers, it's easier to see the memory cells. But "transformers are RNNs" even if it's a little harder to see.

by simonh 4y ago

MonkeyMalarky 4y ago

This is my understanding as well. The amount of mystical projection by users and journalists when talking about it has been astounding. I guess the same could be said of Bard and that one googler who lost their mind and got fired. I guess we'll have to teach future children to "not project real emotions on to the simula" alongside "just because you read it on the internet doesn't mean it's true".

simonh 4y ago
Well, I'm thinking I must be missing some nuance, it seems like there must be some session state stored in the LLM somehow, otherwise it would have to re-parse the entire chat session from scratch to generate each succeeding token. But maybe it does do that. I'm trying to understand if it is learning, or building a model of the world in any persistent way how that is happening and where it exists, if at all.
hrunt 4y ago
"Please don't anthropomorphize the AI. It doesn't like that."
- MonkeyMalarky 4y ago
  Given how delusional people can be, I expect we'll have billboards from People for the Ethical Treatment of Artificial Intelligences (PETAi) with messages like "You wouldn't `systemctl halt` a human child..." by the year 2035.
  - bheadmaster 4y ago
    It's funny that you call them delusional, when we both know that the problem of AI consciousness is unsolved (and might even be unsolvable). I, for one, would most likely become a member of PETAi as soon as I was convinced, without reasonable doubt, that AI had true consciousness like me. I don't see any fundamental reason why matrix multiplication couldn't serve as a vessel to consciousness, but a bunch of chemical reactions in neurons could.

ibejoeb 4y ago

You're basically describing inference, which is using the model to get an output. You're correct that that has no impact on the parameters of the model. One way to arrive at a chat-like "chain of thought" experience is to iteratively build up a prompt based on the previous interactions and feed that in as a new, big prompt on each successive turn.

GistNoesis 4y ago

It's more subtle than that : weights don't change, but the internal state does change at each new word (which allows it to perform few shots learning such as taking and following instructions given in a prompt). This internal state is defined implicitly by the weights that don't change applied to the prompt and generated up to now text (which does change). This internal state is quite big ("dim of the features" times "number of layers" times "current length of text"), and represent an expended view of the relevant context of the conversation, that include things both low level features like what's the previous word, and higher level features like what's the tone and mood of the conversation and what's the direction this conversation is aiming at, so that you can predict what the next word should be accurately. This allow to have "chain of thought" reasoning. This sequence of internal states maybe can be seen as a form of proto stream of consciousness, but this is a controversial opinion to hold. Everything usually is not persisted, except for the conversation themselves that will almost certainly be used in subsequent training datasets where they will be used to improve further models. To keep anthropomorphising, it's like you have plenty of independent chat sessions everyday where you are free to bounce thoughts as they spring into your mind. During the night, some external entity (which are often other models that have been trained to distinguish good from bad) evaluate which conversations were good and which conversations were bad, and make you dream and replay the conversation you had during the day and to updates your world model weights. And the next day you woke up with a tendency to produce better conversations, because of the conversation of yesterday.

simonh 4y ago
Thank you very much, that's great. So there is a mutable working memory for the session that's persistent through the session. That's something that had been missing from previous explanations I'd seen.
- GistNoesis 4y ago
  This mutable working memory is implicit. It comes from the transformer architecture. What the neural network weight are encoding is the dynamic of this implicit working memory such that it produces the desired output. If you use something like LSTM instead of transformers, it's easier to see the memory cells. But "transformers are RNNs" even if it's a little harder to see.