3 ms·

"The mountain in front of me" was a mountain, and I was a mountain. I was a mountain. I was a mountain. I was a mountain. I was a mountain.

• andy99 3y ago
  That kind of repetition is very common in smaller / less powerful models. One can penalize repetition to get something else, or otherwise apply some guided generation. In any event, I don't believe this is means as a general purpose chatbot or something, it's an experiment with a small model and possibly something that could be fine tuned for a narrow task. There's not much point comparing it to much bigger models.
• numeri 3y ago
  Almost all LLM inference these days includes a repetition penalty, to help prevent models from falling into this pattern, the so-called "boredom trap" [1]. Once the model repeats itself once, that greatly increases the chances that the next best token is another repetition (for a raw model like this, at least, that hasn't been chat/instruct/RLHF fine-tuned). I ran the same prompt with a repetition penalty of 1.1 and got this: The mountain in front of me was a little higher than the other two, and I could see the top of the mountain. I'm going to climb it," I said. "It's a good place for a picnic." You're not going to climb it?" No!" I said. "But I'll climb it if you want! I can climb it with you—if we can get out there together—" [1] https://arxiv.org/abs/2007.14966 https://arxiv.org/abs/2007.14966
• Erem 3y ago
  A truly terrible completion but I appreciate it as a poem