Yes, one distribution per position. This was a key innovation that allowed training over the entire sequence in parallel, rather than on one token prediction at

Yes, one distribution per position. This was a key innovation that allowed training over the entire sequence in parallel, rather than on one token prediction at a time, thereby massively speeding up training. More recently, there are models like RWKV that can run in both parallel (GPT-like) mode for training and serial (RNN-like) mode for inference. But transformers always output a probability distribution at each position in the context.