49 ms·

I love this visualization, but when it generates tokens with temperature=1 it seems to be picking the second highest logit/probability, not the highest. Is that

• magicalhippo 2y ago
  It's not explicitly mentioned, but they do implicitly say it here: > We then apply the softmax function to convert the logits into a probability distribution that sums to one. To go from a probability distribution to a single token, you need to sample it, ie pick a random element according to the probability distribution[1]. Elements with higher probability have a higher chance to get picked, but it's still random. This is also why the "temperature" parameter works the way it does. [1]: https://en.wikipedia.org/wiki/Non-uniform_random_variate_generation#Finite_discrete_distributions https://en.wikipedia.org/wiki/Non-uniform_random_variate_gen...
  • jebarker 2y ago
    Ah, I guess the examples are canned for efficiency since it picks the same token each time. Thanks for clarifying.
    • magicalhippo 2y ago
      Hmm, if I select the same example multiple times from the drop-down, it does pick a different output value each time. So seems it does implement the sampling "properly" as such.
      • jebarker 2y ago
        You're right - if you select it from the drop-down it works that way. If you refresh the browser, as I was, it defaults to the same sample.
        • magicalhippo 2y ago
          Ah yes, probably using a fixed seed value for the pseudo-random generator then, so the sequence is fixed as such, but the individual numbers are not correlated. Not unusual.