5 ms·

How does one find out what the max input size is?

by tlonny 4y ago

BoorishBears 4y ago
Search <insert model name> context window size. For LLaMA it's 2048 tokens
- throwayyy479087 4y ago
  Where a token is roughly equal a word?
  - famouswaffles 4y ago
    one token is about 3/4 words
  - axutio 4y ago
    1,000 tokens ≈ 750 words
- MuffinFlavored 4y ago
  what is it for chatgpt?
  - atestu 4y ago
    4k for gpt-3.5 and 8k-32k for gpt4 (there are multiple versions: https://platform.openai.com/docs/models/gpt-4 https://platform.openai.com/docs/models/gpt-4)
    - alchemist1e9 4y ago
      Are you sure there isn’t some additional tricks? Is it possible they are doing summarization when you use the web interface to gpt4? Because I have some extremely long conversations and I ask it if it remembers the beginning of the conversation and it does.
      - dragonwriter 4y ago
        32k tokens is a novella to short novel length, so it can handle an extremely long conversation, but, sure, its possible they are using summarization and/or storage-and-retrieval hacks to extend beyond what fits in the context space naively.
      - d4rkp4ttern 4y ago
        Somehow I doubt there is any type of “prompt compression” being done.
        BoorishBears 4y ago
        It's not an uncommon technique so I'm not sure why you're so doubtful.
        d4rkp4ttern 4y ago
        Because it introduces a possibly recursive application of the model itself and is a bit messy in that there would be some loss of information. It certainly would be a good technique for external applications, though. Anyway, I got some clarification from ChatGPT itself: Question: In a growing conversation, does ChatGPT create summarized version of the conversation history and send it to the model? Answer: In a growing conversation, ChatGPT does not create a summarized version of the entire conversation history and send it to the model. Instead, the model only uses the most recent turns of the conversation to generate a response to the current turn. However, some implementations of ChatGPT do use a technique called "history trimming" to limit the size of the context window that the model considers. History trimming involves discarding some of the oldest turns of the conversation to keep the context window within a manageable size. This allows the model to focus on the most recent turns of the conversation, which may be more relevant to the current turn. For example, OpenAI's GPT-3 API includes a feature called "context chunking" that automatically trims the conversation history to a maximum length of 2048 tokens. If the conversation history exceeds this length, the oldest turns are discarded to keep the context window within the maximum length. Overall, while ChatGPT does not create a summarized version of the entire conversation history, some implementations may use history trimming to limit the size of the context window and improve the model's performance.

How does one find out what the max input size is?

by tlonny 4y ago

How does one find out what the max input size is?

BoorishBears 4y ago

Search <insert model name> context window size. For LLaMA it's 2048 tokens

throwayyy479087 4y ago
Where a token is roughly equal a word?
- famouswaffles 4y ago
  one token is about 3/4 words
- axutio 4y ago
  1,000 tokens ≈ 750 words
MuffinFlavored 4y ago
what is it for chatgpt?
- atestu 4y ago
  4k for gpt-3.5 and 8k-32k for gpt4 (there are multiple versions: https://platform.openai.com/docs/models/gpt-4 https://platform.openai.com/docs/models/gpt-4)
  - alchemist1e9 4y ago
    Are you sure there isn’t some additional tricks? Is it possible they are doing summarization when you use the web interface to gpt4? Because I have some extremely long conversations and I ask it if it remembers the beginning of the conversation and it does.
    - dragonwriter 4y ago
      32k tokens is a novella to short novel length, so it can handle an extremely long conversation, but, sure, its possible they are using summarization and/or storage-and-retrieval hacks to extend beyond what fits in the context space naively.
    - d4rkp4ttern 4y ago
      Somehow I doubt there is any type of “prompt compression” being done.
      - BoorishBears 4y ago
        It's not an uncommon technique so I'm not sure why you're so doubtful.
        d4rkp4ttern 4y ago
        Because it introduces a possibly recursive application of the model itself and is a bit messy in that there would be some loss of information. It certainly would be a good technique for external applications, though. Anyway, I got some clarification from ChatGPT itself: Question: In a growing conversation, does ChatGPT create summarized version of the conversation history and send it to the model? Answer: In a growing conversation, ChatGPT does not create a summarized version of the entire conversation history and send it to the model. Instead, the model only uses the most recent turns of the conversation to generate a response to the current turn. However, some implementations of ChatGPT do use a technique called "history trimming" to limit the size of the context window that the model considers. History trimming involves discarding some of the oldest turns of the conversation to keep the context window within a manageable size. This allows the model to focus on the most recent turns of the conversation, which may be more relevant to the current turn. For example, OpenAI's GPT-3 API includes a feature called "context chunking" that automatically trims the conversation history to a maximum length of 2048 tokens. If the conversation history exceeds this length, the oldest turns are discarded to keep the context window within the maximum length. Overall, while ChatGPT does not create a summarized version of the entire conversation history, some implementations may use history trimming to limit the size of the context window and improve the model's performance.