Not an expert by any means, however I wonder if there's a sliding tail of tokens it uses to form its next words. For example, suppose it uses the last X tokens

Not an expert by any means, however I wonder if there's a sliding tail of tokens it uses to form its next words. For example, suppose it uses the last X tokens to formulate the next word. After a while, all of the previous X tokens are repeated "apologize" tokens, and it loses the plot. You can observe similarly unusual behavior by copying the repeated "apologize" string into a new chat, although responses are much more concise. I understand GPT4 accepts a much larger number of input tokens, if you have access, what happens if you ask for way more repetitions?