5 ms·
I don't understand how it relates to temperature? Are we talking about the temperature parameter that you give LLMs, which for GPT for example is from 0 to 2, w
by mewpmewp2 2y ago
I don't understand how it relates to temperature? Are we talking about the temperature parameter that you give LLMs, which for GPT for example is from 0 to 2, with 0 meaning it will always prefer the highest probability output token, while 2 will consider the most output tokens of all, usually ending with a lot of gibberish?
E.g. if I write "I have a cat and a "
It would have highest probability of picking a word "dog" next, so temperature 0 means it will pretty much always pick dog. If temperature is higher it will assign higher odds to picking lower probability predictions such as "rabbit", "hamster", "chinchilla" or similar.
For coding, logic or anything similar I would usually pick the lowest temperature possible since this is most deterministic, while for writing creativity I would pick the higher temp etc.
- ActorNightly 2y agoIm saying temperature is a hack to make the models actually produce real answers.
- mewpmewp2 2y agoBut they can get also real answers even if you have the temperature setting as 0, where it will always pick the highest scoring token?
- buneskamin 2y agoI think he's saying if you set temp to 0 and answers become deterministic, it will appear that the model is just memorising and reciting. The randomness is a hack that 'forces' the model to generalise by deliberately knocking it off the track of the most probable next token