3 ms·
Roughly analogous process to the temperature parameter in simulated annealing, I think. Low temperature equates to a greedy algorithm (i.e. always take the most
by taylorius 4y ago
Roughly analogous process to the temperature parameter in simulated annealing, I think. Low temperature equates to a greedy algorithm (i.e. always take the most likely next word fragment) - higher temps. increase the probability of taking a lower probability fragment.
- denysvitali 4y agoBasically exploration vs exploitation
- albystein 4y agoI’m not sure if comparing it to the greedy algorithm is the correct way to think about it. Even after applying the temperature, if you use greedy decoding, you’ll still pick the output that has the highest probability. Where the concept of temperature is effective though is when you use a sampling method like sampling from a multinomial distribution, nucleus sampling, etc.
- windsignaling 4y agoThe analogy is apt. In SA, the temperature is used to make the distribution sharper or fatter. In this article, the temperature is used to make the distribution sharper or fatter.