4 ms·
There's no such thing as "base models have only the temperature setting". Models do not have sampling settings (temperature, repetition penalty, etc), the sampl
by dchest 3y ago
There's no such thing as "base models have only the temperature setting". Models do not have sampling settings (temperature, repetition penalty, etc), the sampling code does, which obviously you can use on any model.
For example, here's a function from llama.cpp that applies repetition penalty: https://github.com/ggerganov/llama.cpp/blob/master/llama.cpp#L4893-L4951 https://github.com/ggerganov/llama.cpp/blob/master/llama.cpp...
Here's the one from transformers:
https://github.com/huggingface/transformers/blob/0a55d9f7376f72ad3ff296d4249840021b03bcc4/src/transformers/generation/logits_process.py#L270 https://github.com/huggingface/transformers/blob/0a55d9f7376...
To summarize how they work: you keep some number of previously generated tokens, and once you get logits that you want to sample a new token from, you find the logits for existing tokens and multiply them by a penalty, thus lowering the probability of the corresponding tokens.
- cubefox 3y agoI don't think such penalties were applied to GPT-2 or even GPT-3, yet they weren't repetitive like that.
- dchest 3y agoYes, they are applied. Here's OpenAI doc which describes how to set various sampling parameters for GPT-3: https://platform.openai.com/docs/api-reference/completions/create https://platform.openai.com/docs/api-reference/completions/c... See presence_penalty and frequency_penalty. Sampling techniques is one of important arts of LLMs, you'll can find a lot of papers on them. In general, smaller are more prone to repetition, but you can get caught in it even with larger models.
- dchest 3y agoTo clarify: I meant that in general they are commonly applied, but in this case they weren't as the author confirmed. The repetition, of course, doesn't happen all the time.
- jackcook 3y agoYes, you're right, I should have mentioned it in the post, but I used pure greedy sampling for the GPT-2 outputs since I couldn't do anything but that for the Apple model. So temperature was set to zero, and there was no repetition penalty.