8 ms·

Is the tl;dr: without sufficient context length, the conversation gets confusing, because the AI can't keep track of the full context of the conversation? It's

by xrd 3y ago

Is the tl;dr: without sufficient context length, the conversation gets confusing, because the AI can't keep track of the full context of the conversation? It's like being in conversation with an Alzheimer's patient, who keeps forgetting the last sentences? So, open source models don't do well, while closed source models do? And, this post documents a way to better measure that important metric?

brucethemoose2 3y ago
LLM's have no short term memory. Their internal state never changes. When you have a conversation with one, you are actually feeding it the entire conversation every time you reply, usually with specific formatting the model is trained to "recognize" as a ongoing conversation, like its continuing a long story. There are some other formatting details, like the initial "instruction" always being preserved at the top as the old conversation gets truncated, and there is some caching done so it doesn't have to encode the entire conversation every time. The context size is how much conversation you can feed them at once. The popular open models were essentially limited to ~2K tokens until recently.
- ftxbro 3y ago
  Yes the entire conversation is fed every time. But also the web interfaces for example on ChatGPT or Bing do this automatically as part of their implementation detail, so the one above you is exactly right.
- Der_Einzige 3y ago
  We already know how to do "short term memory". We can mathematically condense/average embedding representations of text and use these as input (i.e. combine a page of text into the representation space of one or several tokens, textual inversion style). Once embeddings get good enough, this kind of memory will be ubiquitous. Not sure why we are collectively having amnesia about sentence transformers and related ideas It's worth noting that "soft-prompts" were a thing that the coomer crowd was already using with openassistant. Not sure why the mainstream NLP crowd is being so damn slow in implementing stuff like this NLP is shocking undertooled, and I wrote a whole gist complaining about this that got to the front page of HN awhile ago: https://gist.github.com/Hellisotherpeople/45c619ee22aac6865ca4bb328eb58faf https://gist.github.com/Hellisotherpeople/45c619ee22aac6865c...
  - brucethemoose2 3y ago
    Soft prompts and textual inversion, as I understand them, required considerable hardware resources and were effectively made obsolete by low precision LORA training. But the text token merging people do for negative prompts is fast iirc. Not that you are wrong. LLM prompting does seem quite primitive compared to zoo of encoding/prompting methods we have for SD... And some of it is directly applicable to LLMs. I never even thought about that before.
    - Der_Einzige 3y ago
      All of it is directly applicable to LLMs. Textual Inversion/Soft Prompts are not made "obsolete" at all by Loras. Textual Inversion is far faster to train, extremely effective on newer models like SDXL which have better clip encoding, and has neat properties of not messing with the general weights of the rest of the model, which is sometimes desirable.
    - ijk 3y ago
      Soft prompts for language models are still viable, but the most accessible version of open source soft prompt training was a TPU-based implementation that depended on features of Google Colab that were no longer available. You can modify it to run locally on a GPU, though it is a bit slow to train. It provides different advantages when compared to a LoRA.
  - TeMPOraL 3y ago
    I occasionally remark or ask, how is it that LLM APIs are always either text -> text, or at best (for smaller models) text -> embeddings, with the idea being of you stuffing the latter in some database for future lookups - where the most interesting thing to expose should be embeddings -> text. As in, I want to be able to feed the LLM arbitrary embedding vectors - whether they come from averaging embeddings of different prompts, some general optimization pattern (textual inversion style, indeed), or just iteratively shifting the input vector in some direction, walking down the latent space and generating output every step, just to see what happens. I ask that, and I get blank stares, or non-answers like "oh, it's always been available, you can e.g. put text in, get embeddings out". Yeah, but I want to do the reverse. It's like I was making so big a category error, asking such nonsensical question, that no one can even parse it. But am I?
- skissane 3y ago
  > LLM's have no short term memory. Their internal state never changes. Many have interpreted recent advances in LLM as a sign that strong AI is just around the corner. Yet, their static nature is an obvious blocker to ever achieving that – humans, even animals, learn continuously, unlike LLMs where training and runtime are two completely separate processes. Which makes me wonder – how difficult would it be to have something like an LLM which continuously learnt just like humans do? Could it be done by modifying existing LLM architectures such as Transformers, or would it require a completely different architecture?
  - anon291 3y ago
    It's actually really easy. Recurrent neural networks have been shown to be totally capable of handling language modeling. For example, rwkv has pretty good recall on par with human short term memory (in my opinion). The issue is training. If you train rwkv like an Rnn it's really expensive and time consuming. If you train it like a gpt.. then it's tractable The math works the same either way. Keep in mind... Llms learn orders of magnitude faster than humans due to parallel nature of transformers.
    - skissane 3y ago
      I don't think that's it though – in recurrent neural networks, there is still a distinction between internal state (dynamic) and weights (static). Whereas, in biological brains, the weights are updated continuously.
      - anon291 3y ago
        There is also a distinction in the brain between long-term and short-term memories. It is not so beyond belief that the brain stores short-term memories into long-term memories in a separate process (Perhaps sleep, whose lack thereof we know is linked with memory recall issues). Recurrent neural networks 'learn' continuously by changing their weights (In effect) based upon the 'previous state'. There are papers showing that attention mechanisms in transformers basically provide a 'weight' update function so that models can 'learn' to accomplish a task based on a few examples. In other words, transformer networks 'learn' to train themselves based on the examples given. It is not so beyond beliefs that recurrent neural networks do the same thing. They learn to set up their internal states such that future tasks can be affected by previous input and patterns. In fact... playing around with models like RWKV, you soon learn that this is a fundamental part of the model. If you start talking to it in a certain way, it will start echoing that back. Clearly it's 'learning'.
- magicalhippo 3y ago
  Why truncate the conversation? Why not have the LLM generate a summary, and use that instead of a truncated conversation? That seems closer to what we do, we recall the important bits without every single word said.
  - brucethemoose2 3y ago
    This is not a bad idea. Some roleplaying UIs kinda do this. But its also several passes through the LLM instead of one, and is prone to distortion.
- regularfry 3y ago
  There's a hack here though, right? Run the LLM once to say "summarise this conversation in 500 words", then tack the new exchange on the end and run it again?
- VMG 3y ago
  Wow, somehow I have never heard about this before. In one sense, this makes LLMs even more impressive - the time it takes to consume a large amount of input seems very small.

by xrd 3y ago

brucethemoose2 3y ago

LLM's have no short term memory. Their internal state never changes. When you have a conversation with one, you are actually feeding it the entire conversation every time you reply, usually with specific formatting the model is trained to "recognize" as a ongoing conversation, like its continuing a long story. There are some other formatting details, like the initial "instruction" always being preserved at the top as the old conversation gets truncated, and there is some caching done so it doesn't have to encode the entire conversation every time. The context size is how much conversation you can feed them at once. The popular open models were essentially limited to ~2K tokens until recently.

ftxbro 3y ago
Yes the entire conversation is fed every time. But also the web interfaces for example on ChatGPT or Bing do this automatically as part of their implementation detail, so the one above you is exactly right.
Der_Einzige 3y ago
We already know how to do "short term memory". We can mathematically condense/average embedding representations of text and use these as input (i.e. combine a page of text into the representation space of one or several tokens, textual inversion style). Once embeddings get good enough, this kind of memory will be ubiquitous. Not sure why we are collectively having amnesia about sentence transformers and related ideas It's worth noting that "soft-prompts" were a thing that the coomer crowd was already using with openassistant. Not sure why the mainstream NLP crowd is being so damn slow in implementing stuff like this NLP is shocking undertooled, and I wrote a whole gist complaining about this that got to the front page of HN awhile ago: https://gist.github.com/Hellisotherpeople/45c619ee22aac6865ca4bb328eb58faf https://gist.github.com/Hellisotherpeople/45c619ee22aac6865c...
- brucethemoose2 3y ago
  Soft prompts and textual inversion, as I understand them, required considerable hardware resources and were effectively made obsolete by low precision LORA training. But the text token merging people do for negative prompts is fast iirc. Not that you are wrong. LLM prompting does seem quite primitive compared to zoo of encoding/prompting methods we have for SD... And some of it is directly applicable to LLMs. I never even thought about that before.
  - Der_Einzige 3y ago
    All of it is directly applicable to LLMs. Textual Inversion/Soft Prompts are not made "obsolete" at all by Loras. Textual Inversion is far faster to train, extremely effective on newer models like SDXL which have better clip encoding, and has neat properties of not messing with the general weights of the rest of the model, which is sometimes desirable.
  - ijk 3y ago
    Soft prompts for language models are still viable, but the most accessible version of open source soft prompt training was a TPU-based implementation that depended on features of Google Colab that were no longer available. You can modify it to run locally on a GPU, though it is a bit slow to train. It provides different advantages when compared to a LoRA.
- TeMPOraL 3y ago
  I occasionally remark or ask, how is it that LLM APIs are always either text -> text, or at best (for smaller models) text -> embeddings, with the idea being of you stuffing the latter in some database for future lookups - where the most interesting thing to expose should be embeddings -> text. As in, I want to be able to feed the LLM arbitrary embedding vectors - whether they come from averaging embeddings of different prompts, some general optimization pattern (textual inversion style, indeed), or just iteratively shifting the input vector in some direction, walking down the latent space and generating output every step, just to see what happens. I ask that, and I get blank stares, or non-answers like "oh, it's always been available, you can e.g. put text in, get embeddings out". Yeah, but I want to do the reverse. It's like I was making so big a category error, asking such nonsensical question, that no one can even parse it. But am I?
skissane 3y ago
> LLM's have no short term memory. Their internal state never changes. Many have interpreted recent advances in LLM as a sign that strong AI is just around the corner. Yet, their static nature is an obvious blocker to ever achieving that – humans, even animals, learn continuously, unlike LLMs where training and runtime are two completely separate processes. Which makes me wonder – how difficult would it be to have something like an LLM which continuously learnt just like humans do? Could it be done by modifying existing LLM architectures such as Transformers, or would it require a completely different architecture?
- anon291 3y ago
  It's actually really easy. Recurrent neural networks have been shown to be totally capable of handling language modeling. For example, rwkv has pretty good recall on par with human short term memory (in my opinion). The issue is training. If you train rwkv like an Rnn it's really expensive and time consuming. If you train it like a gpt.. then it's tractable The math works the same either way. Keep in mind... Llms learn orders of magnitude faster than humans due to parallel nature of transformers.
  - skissane 3y ago
    I don't think that's it though – in recurrent neural networks, there is still a distinction between internal state (dynamic) and weights (static). Whereas, in biological brains, the weights are updated continuously.
    - anon291 3y ago
      There is also a distinction in the brain between long-term and short-term memories. It is not so beyond belief that the brain stores short-term memories into long-term memories in a separate process (Perhaps sleep, whose lack thereof we know is linked with memory recall issues). Recurrent neural networks 'learn' continuously by changing their weights (In effect) based upon the 'previous state'. There are papers showing that attention mechanisms in transformers basically provide a 'weight' update function so that models can 'learn' to accomplish a task based on a few examples. In other words, transformer networks 'learn' to train themselves based on the examples given. It is not so beyond beliefs that recurrent neural networks do the same thing. They learn to set up their internal states such that future tasks can be affected by previous input and patterns. In fact... playing around with models like RWKV, you soon learn that this is a fundamental part of the model. If you start talking to it in a certain way, it will start echoing that back. Clearly it's 'learning'.
magicalhippo 3y ago
Why truncate the conversation? Why not have the LLM generate a summary, and use that instead of a truncated conversation? That seems closer to what we do, we recall the important bits without every single word said.
- brucethemoose2 3y ago
  This is not a bad idea. Some roleplaying UIs kinda do this. But its also several passes through the LLM instead of one, and is prone to distortion.
regularfry 3y ago
There's a hack here though, right? Run the LLM once to say "summarise this conversation in 500 words", then tack the new exchange on the end and run it again?
VMG 3y ago
Wow, somehow I have never heard about this before. In one sense, this makes LLMs even more impressive - the time it takes to consume a large amount of input seems very small.