> A higher temperature θ “excites” previously low probability outputs. I found this misleading, because the low probability outputs were already excited. Consi

> A higher temperature θ “excites” previously low probability outputs. I found this misleading, because the low probability outputs were already excited. Consider that the first softmax formula has no theta in the denominator, so it’s equivalent to theta=1 (max temperature). The way I explained temperature to a colleague last week was: temperature of one results in drawing from the posterior distribution, while temperature of zero gives the argmax that distribution. Intermediate temperatures get some blend between them. But now I see that that hand-wavy description doesn’t explain what happens to the runner ups to the argmax, which this site beautifully does.