No, it doesn't pick a single token based on a set of inputs. It predicts a probability distribution for the next token given the previous tokens. That's why tec

No, it doesn't pick a single token based on a set of inputs. It predicts a probability distribution for the next token given the previous tokens. That's why techniques like beam search and Viterbi work so well - you don't have to commit to the next token at each step. And temperature (what I assume you mean by "randomness injected") isn't "window dressing," it fundamentally gives better results because LMs model probability distributions. You'll get crappy results with any probability model if you run them purely greedily. And you're also neglecting non-causal LMs (like BERT, and encoders in general), which don't predict the next token in a series, but instead predict previous masked tokens. You're conflating how LLMs are used for generation with what LLMs are, and that's just plain wrong. They're not trained autoregressively at all! To repeat, the generation mechanism is simply not part of the LLM. The LLM is a probability model; the generator just uses that model. It's not "breaking it" to use a different generation strategy than greedy autoregression, since they're not even trained a token at a time.