"Short enough sample" is definitely key. With a small model like llama 13B, you can watch the output degrade in real time as it gets longer. When the only thing

"Short enough sample" is definitely key. With a small model like llama 13B, you can watch the output degrade in real time as it gets longer. When the only thing in its context is 1000 words of human prompt, the next LLM-predicted sentence will probably be acceptable. But by the time it's added a few hundred words of its own output to the context, it's going downhill fast. Bigger models slow the rate of degradation, but that's it.