I thought it was common knowledge that LLMs have a context of X tokens (8192 in the case of GPT4) and use those as input to predict the next token probabilities

I thought it was common knowledge that LLMs have a context of X tokens (8192 in the case of GPT4) and use those as input to predict the next token probabilities. Still repetition is a problem, that's why they introduce a penalty for generating the exact same token, so when picking the actual token to output from a list of probabilities, they penalize the ones already in the context so its less likely to repeat itself.