4 ms·

LLM's have no short term memory. Their internal state never changes. When you have a conversation with one, you are actually feeding it the entire conversation

LLM's have no short term memory. Their internal state never changes. When you have a conversation with one, you are actually feeding it the entire conversation every time you reply, usually with specific formatting the model is trained to "recognize" as a ongoing conversation, like its continuing a long story. There are some other formatting details, like the initial "instruction" always being preserved at the top as the old conversation gets truncated, and there is some caching done so it doesn't have to encode the entire conversation every time. The context size is how much conversation you can feed them at once. The popular open models were essentially limited to ~2K tokens until recently.

ftxbro 3y ago
Yes the entire conversation is fed every time. But also the web interfaces for example on ChatGPT or Bing do this automatically as part of their implementation detail, so the one above you is exactly right.
Der_Einzige 3y ago
We already know how to do "short term memory". We can mathematically condense/average embedding representations of text and use these as input (i.e. combine a page of text into the representation space of one or several tokens, textual inversion style). Once embeddings get good enough, this kind of memory will be ubiquitous. Not sure why we are collectively having amnesia about sentence transformers and related ideas It's worth noting that "soft-prompts" were a thing that the coomer crowd was already using with openassistant. Not sure why the mainstream NLP crowd is being so damn slow in implementing stuff like this NLP is shocking undertooled, and I wrote a whole gist complaining about this that got to the front page of HN awhile ago: https://gist.github.com/Hellisotherpeople/45c619ee22aac6865ca4bb328eb58faf https://gist.github.com/Hellisotherpeople/45c619ee22aac6865c...
- brucethemoose2 3y ago
  Soft prompts and textual inversion, as I understand them, required considerable hardware resources and were effectively made obsolete by low precision LORA training. But the text token merging people do for negative prompts is fast iirc. Not that you are wrong. LLM prompting does seem quite primitive compared to zoo of encoding/prompting methods we have for SD... And some of it is directly applicable to LLMs. I never even thought about that before.
  - Der_Einzige 3y ago
    All of it is directly applicable to LLMs. Textual Inversion/Soft Prompts are not made "obsolete" at all by Loras. Textual Inversion is far faster to train, extremely effective on newer models like SDXL which have better clip encoding, and has neat properties of not messing with the general weights of the rest of the model, which is sometimes desirable.
  - ijk 3y ago
    Soft prompts for language models are still viable, but the most accessible version of open source soft prompt training was a TPU-based implementation that depended on features of Google Colab that were no longer available. You can modify it to run locally on a GPU, though it is a bit slow to train. It provides different advantages when compared to a LoRA.
- TeMPOraL 3y ago
  I occasionally remark or ask, how is it that LLM APIs are always either text -> text, or at best (for smaller models) text -> embeddings, with the idea being of you stuffing the latter in some database for future lookups - where the most interesting thing to expose should be embeddings -> text. As in, I want to be able to feed the LLM arbitrary embedding vectors - whether they come from averaging embeddings of different prompts, some general optimization pattern (textual inversion style, indeed), or just iteratively shifting the input vector in some direction, walking down the latent space and generating output every step, just to see what happens. I ask that, and I get blank stares, or non-answers like "oh, it's always been available, you can e.g. put text in, get embeddings out". Yeah, but I want to do the reverse. It's like I was making so big a category error, asking such nonsensical question, that no one can even parse it. But am I?
  - PeterisP 3y ago
    What exactly do you mean by "embeddings" there? The way a transformer-based LLM works is that right after tokenization you get a "vector of embedding vectors" i.e. an embedding for each token, and that gets processed by the actual transformer layers. If instead of a list of tokens you want to pass in a list of embedding vectors, that is trivial (assuming you're running a LLM with your own code locally), that would be a <5 line function. However, I'm not really sure when/why I would want to do that, as "averaging embeddings of different prompts, some general optimization pattern (textual inversion style, indeed), or just iteratively shifting the input vector in some direction, walking down the latent space and generating output every step" are NOT really enabled by that - since that inherently must be a vector of token-embedding vectors combined with positional embeddings, you can't just average different prompts, and there isn't a clear way how to tweak the individual token embeddings that would map to tweaking the whole prompt in some meaningful direction. In essence, your question seems to make sense if the text was converted to a single "embedding vector", but it's not, at every layer of a transformer model the state is a list of separate embeddings distinctly separated at a token level.
    - TeMPOraL 3y ago
      > If instead of a list of tokens you want to pass in a list of embedding vectors, that is trivial (assuming you're running a LLM with your own code locally), that would be a <5 line function. That sounds like what I wanted. I haven't got to the point of running open-source LLMs locally yet, so I didn't realize it's as easy as you say - rather, since I haven't seen it being mentioned as something you do with LLM, I assumed it must be nontrivial. > However, I'm not really sure when/why I would want to do that High-level dream: I want to explore the latent space of an LLM. Get a feel for how it's organized, and what of it is spurious, and what structures seem to be common among LLMs. Find answers to questions that will come to me when I "go there" and "look around". It might be a futile pursuit - but I feel it's not the idea that's the problem, but rather that high-dimensional spaces might be too big to explore without mathematical understanding that's way beyond mine. > as "[the specific ideas I listed]" are NOT really enabled by that - since that inherently must be a vector of token-embedding vectors combined with positional embeddings, you can't just average different prompts, In general, or at all? Am I wrong in suspecting that averaging the embeddings of "orange" and "apple" and "banana" may give us a new embedding that, fed to the LLM, will make it output something meaningful (vs. random)? From what I read today, elsewhere in the thread, it seems that this works as expected for Stable Diffusion family of models. And sure, if I take two random multi-token prompts, I expect the result will make LLM output some noise in the shape of word salad. But what if it's multiple closely-related prompts? I have my suspicions, but I want to find out for sure. > and there isn't a clear way how to tweak the individual token embeddings that would map to tweaking the whole prompt in some meaningful direction. Shifting prompts in some meaningful direction may not be trivial (or useful), but it's not the only interesting thing to do. What if I push a single token in the prompt a little bit in some direction? What if I push all of them? How things change with subsequent pushes? > In essence, your question seems to make sense if the text was converted to a single "embedding vector", but it's not, at every layer of a transformer model the state is a list of separate embeddings distinctly separated at a token level. I'm definitely not assuming the text can be converted to a single "embedding vector". My questions boil down to, where is debug-level access to the LLM? Why is everyone focused on stringing tokens, instead of looking at what's lurking between the tokens and their sequences, in areas of the latent space you can't easily get to without embeddings that don't map to any possible prompt. IDK, maybe my mental model of transformer models is completely broken, or maybe all those questions have been asked, and didn't lead to anything of interest, and I just can't seem to find the right papers covering this.
skissane 3y ago
> LLM's have no short term memory. Their internal state never changes. Many have interpreted recent advances in LLM as a sign that strong AI is just around the corner. Yet, their static nature is an obvious blocker to ever achieving that – humans, even animals, learn continuously, unlike LLMs where training and runtime are two completely separate processes. Which makes me wonder – how difficult would it be to have something like an LLM which continuously learnt just like humans do? Could it be done by modifying existing LLM architectures such as Transformers, or would it require a completely different architecture?
- anon291 3y ago
  It's actually really easy. Recurrent neural networks have been shown to be totally capable of handling language modeling. For example, rwkv has pretty good recall on par with human short term memory (in my opinion). The issue is training. If you train rwkv like an Rnn it's really expensive and time consuming. If you train it like a gpt.. then it's tractable The math works the same either way. Keep in mind... Llms learn orders of magnitude faster than humans due to parallel nature of transformers.
  - skissane 3y ago
    I don't think that's it though – in recurrent neural networks, there is still a distinction between internal state (dynamic) and weights (static). Whereas, in biological brains, the weights are updated continuously.
    - anon291 3y ago
      There is also a distinction in the brain between long-term and short-term memories. It is not so beyond belief that the brain stores short-term memories into long-term memories in a separate process (Perhaps sleep, whose lack thereof we know is linked with memory recall issues). Recurrent neural networks 'learn' continuously by changing their weights (In effect) based upon the 'previous state'. There are papers showing that attention mechanisms in transformers basically provide a 'weight' update function so that models can 'learn' to accomplish a task based on a few examples. In other words, transformer networks 'learn' to train themselves based on the examples given. It is not so beyond beliefs that recurrent neural networks do the same thing. They learn to set up their internal states such that future tasks can be affected by previous input and patterns. In fact... playing around with models like RWKV, you soon learn that this is a fundamental part of the model. If you start talking to it in a certain way, it will start echoing that back. Clearly it's 'learning'.
magicalhippo 3y ago
Why truncate the conversation? Why not have the LLM generate a summary, and use that instead of a truncated conversation? That seems closer to what we do, we recall the important bits without every single word said.
- brucethemoose2 3y ago
  This is not a bad idea. Some roleplaying UIs kinda do this. But its also several passes through the LLM instead of one, and is prone to distortion.
regularfry 3y ago
There's a hack here though, right? Run the LLM once to say "summarise this conversation in 500 words", then tack the new exchange on the end and run it again?
VMG 3y ago
Wow, somehow I have never heard about this before. In one sense, this makes LLMs even more impressive - the time it takes to consume a large amount of input seems very small.

ftxbro 3y ago

Yes the entire conversation is fed every time. But also the web interfaces for example on ChatGPT or Bing do this automatically as part of their implementation detail, so the one above you is exactly right.

Der_Einzige 3y ago

We already know how to do "short term memory". We can mathematically condense/average embedding representations of text and use these as input (i.e. combine a page of text into the representation space of one or several tokens, textual inversion style). Once embeddings get good enough, this kind of memory will be ubiquitous. Not sure why we are collectively having amnesia about sentence transformers and related ideas It's worth noting that "soft-prompts" were a thing that the coomer crowd was already using with openassistant. Not sure why the mainstream NLP crowd is being so damn slow in implementing stuff like this NLP is shocking undertooled, and I wrote a whole gist complaining about this that got to the front page of HN awhile ago: https://gist.github.com/Hellisotherpeople/45c619ee22aac6865ca4bb328eb58faf https://gist.github.com/Hellisotherpeople/45c619ee22aac6865c...

brucethemoose2 3y ago
Soft prompts and textual inversion, as I understand them, required considerable hardware resources and were effectively made obsolete by low precision LORA training. But the text token merging people do for negative prompts is fast iirc. Not that you are wrong. LLM prompting does seem quite primitive compared to zoo of encoding/prompting methods we have for SD... And some of it is directly applicable to LLMs. I never even thought about that before.
- Der_Einzige 3y ago
  All of it is directly applicable to LLMs. Textual Inversion/Soft Prompts are not made "obsolete" at all by Loras. Textual Inversion is far faster to train, extremely effective on newer models like SDXL which have better clip encoding, and has neat properties of not messing with the general weights of the rest of the model, which is sometimes desirable.
- ijk 3y ago
  Soft prompts for language models are still viable, but the most accessible version of open source soft prompt training was a TPU-based implementation that depended on features of Google Colab that were no longer available. You can modify it to run locally on a GPU, though it is a bit slow to train. It provides different advantages when compared to a LoRA.
TeMPOraL 3y ago
I occasionally remark or ask, how is it that LLM APIs are always either text -> text, or at best (for smaller models) text -> embeddings, with the idea being of you stuffing the latter in some database for future lookups - where the most interesting thing to expose should be embeddings -> text. As in, I want to be able to feed the LLM arbitrary embedding vectors - whether they come from averaging embeddings of different prompts, some general optimization pattern (textual inversion style, indeed), or just iteratively shifting the input vector in some direction, walking down the latent space and generating output every step, just to see what happens. I ask that, and I get blank stares, or non-answers like "oh, it's always been available, you can e.g. put text in, get embeddings out". Yeah, but I want to do the reverse. It's like I was making so big a category error, asking such nonsensical question, that no one can even parse it. But am I?
- PeterisP 3y ago
  What exactly do you mean by "embeddings" there? The way a transformer-based LLM works is that right after tokenization you get a "vector of embedding vectors" i.e. an embedding for each token, and that gets processed by the actual transformer layers. If instead of a list of tokens you want to pass in a list of embedding vectors, that is trivial (assuming you're running a LLM with your own code locally), that would be a <5 line function. However, I'm not really sure when/why I would want to do that, as "averaging embeddings of different prompts, some general optimization pattern (textual inversion style, indeed), or just iteratively shifting the input vector in some direction, walking down the latent space and generating output every step" are NOT really enabled by that - since that inherently must be a vector of token-embedding vectors combined with positional embeddings, you can't just average different prompts, and there isn't a clear way how to tweak the individual token embeddings that would map to tweaking the whole prompt in some meaningful direction. In essence, your question seems to make sense if the text was converted to a single "embedding vector", but it's not, at every layer of a transformer model the state is a list of separate embeddings distinctly separated at a token level.
  - TeMPOraL 3y ago
    > If instead of a list of tokens you want to pass in a list of embedding vectors, that is trivial (assuming you're running a LLM with your own code locally), that would be a <5 line function. That sounds like what I wanted. I haven't got to the point of running open-source LLMs locally yet, so I didn't realize it's as easy as you say - rather, since I haven't seen it being mentioned as something you do with LLM, I assumed it must be nontrivial. > However, I'm not really sure when/why I would want to do that High-level dream: I want to explore the latent space of an LLM. Get a feel for how it's organized, and what of it is spurious, and what structures seem to be common among LLMs. Find answers to questions that will come to me when I "go there" and "look around". It might be a futile pursuit - but I feel it's not the idea that's the problem, but rather that high-dimensional spaces might be too big to explore without mathematical understanding that's way beyond mine. > as "[the specific ideas I listed]" are NOT really enabled by that - since that inherently must be a vector of token-embedding vectors combined with positional embeddings, you can't just average different prompts, In general, or at all? Am I wrong in suspecting that averaging the embeddings of "orange" and "apple" and "banana" may give us a new embedding that, fed to the LLM, will make it output something meaningful (vs. random)? From what I read today, elsewhere in the thread, it seems that this works as expected for Stable Diffusion family of models. And sure, if I take two random multi-token prompts, I expect the result will make LLM output some noise in the shape of word salad. But what if it's multiple closely-related prompts? I have my suspicions, but I want to find out for sure. > and there isn't a clear way how to tweak the individual token embeddings that would map to tweaking the whole prompt in some meaningful direction. Shifting prompts in some meaningful direction may not be trivial (or useful), but it's not the only interesting thing to do. What if I push a single token in the prompt a little bit in some direction? What if I push all of them? How things change with subsequent pushes? > In essence, your question seems to make sense if the text was converted to a single "embedding vector", but it's not, at every layer of a transformer model the state is a list of separate embeddings distinctly separated at a token level. I'm definitely not assuming the text can be converted to a single "embedding vector". My questions boil down to, where is debug-level access to the LLM? Why is everyone focused on stringing tokens, instead of looking at what's lurking between the tokens and their sequences, in areas of the latent space you can't easily get to without embeddings that don't map to any possible prompt. IDK, maybe my mental model of transformer models is completely broken, or maybe all those questions have been asked, and didn't lead to anything of interest, and I just can't seem to find the right papers covering this.

skissane 3y ago

> LLM's have no short term memory. Their internal state never changes. Many have interpreted recent advances in LLM as a sign that strong AI is just around the corner. Yet, their static nature is an obvious blocker to ever achieving that – humans, even animals, learn continuously, unlike LLMs where training and runtime are two completely separate processes. Which makes me wonder – how difficult would it be to have something like an LLM which continuously learnt just like humans do? Could it be done by modifying existing LLM architectures such as Transformers, or would it require a completely different architecture?

anon291 3y ago
It's actually really easy. Recurrent neural networks have been shown to be totally capable of handling language modeling. For example, rwkv has pretty good recall on par with human short term memory (in my opinion). The issue is training. If you train rwkv like an Rnn it's really expensive and time consuming. If you train it like a gpt.. then it's tractable The math works the same either way. Keep in mind... Llms learn orders of magnitude faster than humans due to parallel nature of transformers.
- skissane 3y ago
  I don't think that's it though – in recurrent neural networks, there is still a distinction between internal state (dynamic) and weights (static). Whereas, in biological brains, the weights are updated continuously.
  - anon291 3y ago
    There is also a distinction in the brain between long-term and short-term memories. It is not so beyond belief that the brain stores short-term memories into long-term memories in a separate process (Perhaps sleep, whose lack thereof we know is linked with memory recall issues). Recurrent neural networks 'learn' continuously by changing their weights (In effect) based upon the 'previous state'. There are papers showing that attention mechanisms in transformers basically provide a 'weight' update function so that models can 'learn' to accomplish a task based on a few examples. In other words, transformer networks 'learn' to train themselves based on the examples given. It is not so beyond beliefs that recurrent neural networks do the same thing. They learn to set up their internal states such that future tasks can be affected by previous input and patterns. In fact... playing around with models like RWKV, you soon learn that this is a fundamental part of the model. If you start talking to it in a certain way, it will start echoing that back. Clearly it's 'learning'.

magicalhippo 3y ago

Why truncate the conversation? Why not have the LLM generate a summary, and use that instead of a truncated conversation? That seems closer to what we do, we recall the important bits without every single word said.

brucethemoose2 3y ago
This is not a bad idea. Some roleplaying UIs kinda do this. But its also several passes through the LLM instead of one, and is prone to distortion.

regularfry 3y ago

There's a hack here though, right? Run the LLM once to say "summarise this conversation in 500 words", then tack the new exchange on the end and run it again?

VMG 3y ago

Wow, somehow I have never heard about this before. In one sense, this makes LLMs even more impressive - the time it takes to consume a large amount of input seems very small.