3 ms·
Transformer Explainer
- mannykannot 2y agoThe article uses the phrase 'semantic meaning' three times, but we are dealing with tokens here, which leads me to wonder what sort of semantics sub-word tokens have. For example, does compositionality [1] apply over sub-word tokens? Does the success of LLMs in generating text that could pass as human-generated suggest that it does? [1] https://en.wikipedia.org/wiki/Principle_of_compositionality https://en.wikipedia.org/wiki/Principle_of_compositionality
- jebarker 2y agoI love this visualization, but when it generates tokens with temperature=1 it seems to be picking the second highest logit/probability, not the highest. Is that a mistake or am I missing something?
- magicalhippo 2y agoIt's not explicitly mentioned, but they do implicitly say it here: > We then apply the softmax function to convert the logits into a probability distribution that sums to one. To go from a probability distribution to a single token, you need to sample it, ie pick a random element according to the probability distribution[1]. Elements with higher probability have a higher chance to get picked, but it's still random. This is also why the "temperature" parameter works the way it does. [1]: https://en.wikipedia.org/wiki/Non-uniform_random_variate_generation#Finite_discrete_distributions https://en.wikipedia.org/wiki/Non-uniform_random_variate_gen...
- jebarker 2y agoAh, I guess the examples are canned for efficiency since it picks the same token each time. Thanks for clarifying.
- magicalhippo 2y agoHmm, if I select the same example multiple times from the drop-down, it does pick a different output value each time. So seems it does implement the sampling "properly" as such.
- jebarker 2y agoYou're right - if you select it from the drop-down it works that way. If you refresh the browser, as I was, it defaults to the same sample.
- magicalhippo 2y agoAh yes, probably using a fixed seed value for the pseudo-random generator then, so the sequence is fixed as such, but the individual numbers are not correlated. Not unusual.
- deleted 2y ago[deleted]
- LZ_Khan 2y agoQuestion: what is the purpose of softmax for words/tokens before the final word? Isn't it only the softmax distribution respective to the final word that gets used for the next token prediction? That was my understanding at least.