This is a common problem with language models in general, and the reason is not that well understood. This paper A Theoretical Analysis of the Repetition Probl

This is a common problem with language models in general, and the reason is not that well understood. This paper A Theoretical Analysis of the Repetition Problem in Text Generation (https://arxiv.org/abs/2012.14660 https://arxiv.org/abs/2012.14660) seems to offer a principled answer. Basically the probability maximizing search procedure for text generation can get stuck in loops where the most likely next statement is similar or same to before. I'm no NLP researcher so I don't have easy intuition on it, but that paper seems like a good read.