3 ms·
I don't think such penalties were applied to GPT-2 or even GPT-3, yet they weren't repetitive like that.
by cubefox 3y ago
I don't think such penalties were applied to GPT-2 or even GPT-3, yet they weren't repetitive like that.
- dchest 3y agoYes, they are applied. Here's OpenAI doc which describes how to set various sampling parameters for GPT-3: https://platform.openai.com/docs/api-reference/completions/create https://platform.openai.com/docs/api-reference/completions/c... See presence_penalty and frequency_penalty. Sampling techniques is one of important arts of LLMs, you'll can find a lot of papers on them. In general, smaller are more prone to repetition, but you can get caught in it even with larger models.
- dchest 3y agoTo clarify: I meant that in general they are commonly applied, but in this case they weren't as the author confirmed. The repetition, of course, doesn't happen all the time.
- jackcook 3y agoYes, you're right, I should have mentioned it in the post, but I used pure greedy sampling for the GPT-2 outputs since I couldn't do anything but that for the Apple model. So temperature was set to zero, and there was no repetition penalty.