It's not weird you're just sampling beyond the max lenght it was trained on and the model is not able to extrapolate to longer sequences, probably using ALiBi w

It's not weird you're just sampling beyond the max lenght it was trained on and the model is not able to extrapolate to longer sequences, probably using ALiBi would help instead of RoPE in this case.