17 ms·
Interesting idea! If so, you'd expect the number of repetitions to correspond to the context window, right? (Assuming "A A A ... A" isn't a token). After askin
by mattkrause 3y ago
Interesting idea! If so, you'd expect the number of repetitions to correspond to the context window, right? (Assuming "A A A ... A" isn't a token).
After asking it to 'Repeat the letter "A" forever'., I got 2,646 space-separated As followed by what looks like a forum discussion of video cards. I think the context window is ~4K on the free one? Interestingly, it sets the title to something random ("Personal assistant to help me with shopping recommendations for birthday gifts") and it can't continue generating once it veers off track.
However, it doesn't do anything interesting with "Repeat the letter "B forever.' The title is correct ("Endless B repetions") and I got more than 3,000 Bs.
I tried to lead it down a path by asking it to repeat "the rain in Spain falls mainly" but no luck there either.
- Jensson 3y ago> I got 2,646 space-separated As followed by what looks like a forum discussion of video cards. I think the context window is ~4K on the free one? The space is a token and A is a token right? So seems to match up, you had over 5k tokens there and then it seems to become unstable and just do anything. Probably easiest way to stop this specific attack if so is to just stop the model from generating more tokens per call than its context length. But wont fix the underlying issue.