What you’ve described is basically the problem with greedy sampling in the decoder. Many other local optimization sampling strategies exist (e.g., beam search)

What you’ve described is basically the problem with greedy sampling in the decoder. Many other local optimization sampling strategies exist (e.g., beam search) and there’s been a lot of work on more global sampling (e.g., speculative decoding).