4 ms·
A probability distribution for each token in the array, not just the last one? I don't understand that, because wouldn't the probabilities later in the sentenc
by flir 4y ago
A probability distribution for each token in the array, not just the last one?
I don't understand that, because wouldn't the probabilities later in the sentence be impacted by the tokens chosen earlier in the sentence?
- ruuda 4y agoYou unfold it one token at a time by sampling from the returned distribution. To control the amount of variation, you can make the probability distribution more extreme, in the most extreme case you only select the most likely token, and the sequence becomes deterministic. Yes, what happens later in the sentence depends on the particular choice you made earlier in the sentence.
- smaddox 4y agoYes, one distribution per position. This was a key innovation that allowed training over the entire sequence in parallel, rather than on one token prediction at a time, thereby massively speeding up training. More recently, there are models like RWKV that can run in both parallel (GPT-like) mode for training and serial (RNN-like) mode for inference. But transformers always output a probability distribution at each position in the context.