3 ms·

Search <insert model name> context window size. For LLaMA it's 2048 tokens

by BoorishBears 4y ago

throwayyy479087 4y ago
Where a token is roughly equal a word?
- famouswaffles 4y ago
  one token is about 3/4 words
- axutio 4y ago
  1,000 tokens ≈ 750 words
MuffinFlavored 4y ago
what is it for chatgpt?
- atestu 4y ago
  4k for gpt-3.5 and 8k-32k for gpt4 (there are multiple versions: https://platform.openai.com/docs/models/gpt-4 https://platform.openai.com/docs/models/gpt-4)
  - alchemist1e9 4y ago
    Are you sure there isn’t some additional tricks? Is it possible they are doing summarization when you use the web interface to gpt4? Because I have some extremely long conversations and I ask it if it remembers the beginning of the conversation and it does.
    - dragonwriter 4y ago
      32k tokens is a novella to short novel length, so it can handle an extremely long conversation, but, sure, its possible they are using summarization and/or storage-and-retrieval hacks to extend beyond what fits in the context space naively.
    - d4rkp4ttern 4y ago
      Somehow I doubt there is any type of “prompt compression” being done.
      - BoorishBears 4y ago
        It's not an uncommon technique so I'm not sure why you're so doubtful.
        d4rkp4ttern 4y ago
        Because it introduces a possibly recursive application of the model itself and is a bit messy in that there would be some loss of information. It certainly would be a good technique for external applications, though. Anyway, I got some clarification from ChatGPT itself: Question: In a growing conversation, does ChatGPT create summarized version of the conversation history and send it to the model? Answer: In a growing conversation, ChatGPT does not create a summarized version of the entire conversation history and send it to the model. Instead, the model only uses the most recent turns of the conversation to generate a response to the current turn. However, some implementations of ChatGPT do use a technique called "history trimming" to limit the size of the context window that the model considers. History trimming involves discarding some of the oldest turns of the conversation to keep the context window within a manageable size. This allows the model to focus on the most recent turns of the conversation, which may be more relevant to the current turn. For example, OpenAI's GPT-3 API includes a feature called "context chunking" that automatically trims the conversation history to a maximum length of 2048 tokens. If the conversation history exceeds this length, the oldest turns are discarded to keep the context window within the maximum length. Overall, while ChatGPT does not create a summarized version of the entire conversation history, some implementations may use history trimming to limit the size of the context window and improve the model's performance.
        Guillaume86 4y ago
        Why would you trust what ChatGPT answered you? Here's what I got out of it: Q: In a growing conversation, does ChatGPT create a summarized version of the conversation history to build the prompt when it's getting too big for its context size? A: Yes, ChatGPT may create a summarized version of the conversation history when it's getting too big for its context size. This is because the GPT-3.5 architecture used by ChatGPT has a limited context size, which means that it can only take into account a certain number of previous messages in the conversation when generating a response. To work around this limitation, ChatGPT may use a summarization algorithm to condense the conversation history into a more manageable size, while still preserving the key points and important information. This summarization process may involve techniques such as text clustering, topic modeling, or text summarization algorithms like BERTSum, T5, or GPT itself. Once the conversation history has been summarized, ChatGPT can use the condensed version as the prompt to generate a response that is more focused and relevant to the current topic of discussion.
    - BoorishBears 4y ago
      You can always implement summarization, langchain for example support this. But that's not changing how many tokens the model itself is consuming

Search <insert model name> context window size. For LLaMA it's 2048 tokens

throwayyy479087 4y ago

Where a token is roughly equal a word?

famouswaffles 4y ago
one token is about 3/4 words
axutio 4y ago
1,000 tokens ≈ 750 words

MuffinFlavored 4y ago

what is it for chatgpt?

atestu 4y ago
4k for gpt-3.5 and 8k-32k for gpt4 (there are multiple versions: https://platform.openai.com/docs/models/gpt-4 https://platform.openai.com/docs/models/gpt-4)
- alchemist1e9 4y ago
  Are you sure there isn’t some additional tricks? Is it possible they are doing summarization when you use the web interface to gpt4? Because I have some extremely long conversations and I ask it if it remembers the beginning of the conversation and it does.
  - dragonwriter 4y ago
    32k tokens is a novella to short novel length, so it can handle an extremely long conversation, but, sure, its possible they are using summarization and/or storage-and-retrieval hacks to extend beyond what fits in the context space naively.
  - d4rkp4ttern 4y ago
    Somehow I doubt there is any type of “prompt compression” being done.
    - BoorishBears 4y ago
      It's not an uncommon technique so I'm not sure why you're so doubtful.
      - d4rkp4ttern 4y ago
        Because it introduces a possibly recursive application of the model itself and is a bit messy in that there would be some loss of information. It certainly would be a good technique for external applications, though. Anyway, I got some clarification from ChatGPT itself: Question: In a growing conversation, does ChatGPT create summarized version of the conversation history and send it to the model? Answer: In a growing conversation, ChatGPT does not create a summarized version of the entire conversation history and send it to the model. Instead, the model only uses the most recent turns of the conversation to generate a response to the current turn. However, some implementations of ChatGPT do use a technique called "history trimming" to limit the size of the context window that the model considers. History trimming involves discarding some of the oldest turns of the conversation to keep the context window within a manageable size. This allows the model to focus on the most recent turns of the conversation, which may be more relevant to the current turn. For example, OpenAI's GPT-3 API includes a feature called "context chunking" that automatically trims the conversation history to a maximum length of 2048 tokens. If the conversation history exceeds this length, the oldest turns are discarded to keep the context window within the maximum length. Overall, while ChatGPT does not create a summarized version of the entire conversation history, some implementations may use history trimming to limit the size of the context window and improve the model's performance.
        Guillaume86 4y ago
        Why would you trust what ChatGPT answered you? Here's what I got out of it: Q: In a growing conversation, does ChatGPT create a summarized version of the conversation history to build the prompt when it's getting too big for its context size? A: Yes, ChatGPT may create a summarized version of the conversation history when it's getting too big for its context size. This is because the GPT-3.5 architecture used by ChatGPT has a limited context size, which means that it can only take into account a certain number of previous messages in the conversation when generating a response. To work around this limitation, ChatGPT may use a summarization algorithm to condense the conversation history into a more manageable size, while still preserving the key points and important information. This summarization process may involve techniques such as text clustering, topic modeling, or text summarization algorithms like BERTSum, T5, or GPT itself. Once the conversation history has been summarized, ChatGPT can use the condensed version as the prompt to generate a response that is more focused and relevant to the current topic of discussion.
  - BoorishBears 4y ago
    You can always implement summarization, langchain for example support this. But that's not changing how many tokens the model itself is consuming