4 ms·

Models don't support context. You feed prior prompt/response pairs back into the input.

• tlonny 4y ago
  How does one find out what the max input size is?
  • BoorishBears 4y ago
    Search <insert model name> context window size. For LLaMA it's 2048 tokens
    • throwayyy479087 4y ago
      Where a token is roughly equal a word?
      • famouswaffles 4y ago
        one token is about 3/4 words
      • axutio 4y ago
        1,000 tokens ≈ 750 words
    • MuffinFlavored 4y ago
      what is it for chatgpt?
      • atestu 4y ago
        4k for gpt-3.5 and 8k-32k for gpt4 (there are multiple versions: https://platform.openai.com/docs/models/gpt-4 https://platform.openai.com/docs/models/gpt-4)
        • alchemist1e9 4y ago
          Are you sure there isn’t some additional tricks? Is it possible they are doing summarization when you use the web interface to gpt4? Because I have some extremely long conversations and I ask it if it remembers the beginning of the conversation and it does.
          • dragonwriter 4y ago
            32k tokens is a novella to short novel length, so it can handle an extremely long conversation, but, sure, its possible they are using summarization and/or storage-and-retrieval hacks to extend beyond what fits in the context space naively.
          • d4rkp4ttern 4y ago
            Somehow I doubt there is any type of “prompt compression” being done.
            • BoorishBears 4y ago
              It's not an uncommon technique so I'm not sure why you're so doubtful.
              • d4rkp4ttern 4y ago
                Because it introduces a possibly recursive application of the model itself and is a bit messy in that there would be some loss of information. It certainly would be a good technique for external applications, though. Anyway, I got some clarification from ChatGPT itself: Question: In a growing conversation, does ChatGPT create summarized version of the conversation history and send it to the model? Answer: In a growing conversation, ChatGPT does not create a summarized version of the entire conversation history and send it to the model. Instead, the model only uses the most recent turns of the conversation to generate a response to the current turn. However, some implementations of ChatGPT do use a technique called "history trimming" to limit the size of the context window that the model considers. History trimming involves discarding some of the oldest turns of the conversation to keep the context window within a manageable size. This allows the model to focus on the most recent turns of the conversation, which may be more relevant to the current turn. For example, OpenAI's GPT-3 API includes a feature called "context chunking" that automatically trims the conversation history to a maximum length of 2048 tokens. If the conversation history exceeds this length, the oldest turns are discarded to keep the context window within the maximum length. Overall, while ChatGPT does not create a summarized version of the entire conversation history, some implementations may use history trimming to limit the size of the context window and improve the model's performance.
• barbazoo 4y ago
  Oh is that how it works under the hood?
  • qeternity 4y ago
    Yes. The chat functionality is really just a clever trick. All LLMs are still just predicting the next token in a sequence (i.e. finishing the sentence).
  • vdfs 4y ago
    It's not a secret, in OpenAI api you have to keep sending previous question and responses on top of your new question, essentially you are asking new question but give it more context with the previous questions and answers
    • d4rkp4ttern 4y ago
      Thank you for clarifying this mystery. My mental model of how ChatGPT works is now clearer. I was somehow thinking that in chat mode it would either (a) need to “update state”, or (b) be fed increasingly longer history. I thought (b) would slow down later responses but I guess with their ginormous compute it’s not perceptible to users.
      • Datagenerator 4y ago
        Someone made a version with a buffer which only has a fixed size context history so it stays in constant speed mode. It will only focus on the latest questions.
  • Veen 4y ago
    Yes, it's why in longer conversations it might forget what was said earlier in the thread. The token limits still apply.
• sterlind 4y ago
  I misspoke. My question is whether the model was fine-tuned on multiple prompt/response pairs (with later prompts referencing earlier ones) or only on one-shot prompts. I expect fine-tuning on e.g.: User: Pick an animal. Bot: Horse User: How many legs does it have? Bot: 4 to be different from just fine-tuning it on User: Pick an animal. Bot: Horse Especially when it comes to e.g. training the bot to ask follow-up questions, etc.