3 ms·

Pretty sure it gets the entire conversation as input

• seydor 4y ago
  Is this described somewhere? Wikipedia doesn't help
  • Miraste 4y ago
    It feeds the entire previous conversation back into the AI until it reaches the token limit. This isn't a new technique; the first place I saw it was in AI Dungeon (an AI writing/adventure app) back on GPT-2. NovelAI, another writing app, has some interesting tweaks to keep a consistent story, like a "lorebook" to associate words with descriptions, which it then swaps into the input when those words come up so the AI can "remember" farther back. I wouldn't be surprised if ChatGPT has some automated version of this in addition to the naive rolling feedback.
    • int_19h 4y ago
      What I find interesting is that, with some complicated tasks, the model will produce output that reiterates some of the key points from earlier, and it can continue to do so for a very long time (i.e. long enough that they'd normally be out of the token window). It feels vaguely like crude short-term memory.
  • macrolime 4y ago
    It preserves context by taking using the whole context as input. On the section "one difference from BERT" here it is described a bit more how this works, including some nice animations. http://jalammar.github.io/illustrated-gpt2/ http://jalammar.github.io/illustrated-gpt2/ It uses both its own output from previous steps and the users prompt(s) as input for each token(word) that it predicts.