The neural network just generates a set of probabilities for all tokens. The actual next token is then sampled from this set, which is always random for T>0 (an

The neural network just generates a set of probabilities for all tokens. The actual next token is then sampled from this set, which is always random for T>0 (and in the case of GPT4 even for T=0, because of the way the model itself works).