The problem is that a position's probability output is conditioned via attention on all previous positions. If you want to be better you need to switch to DDPM

The problem is that a position's probability output is conditioned via attention on all previous positions. If you want to be better you need to switch to DDPMs for example (e.g. an encoder-only transformer to predict diffusion transition probabilities in parallel, then apply steps of denoising). The problem is just that these don't work so well from auto regressive decoder transformers, and encoder-decoder architectures like e.g. Google's T5 have fallen out of favor since about LLAMA dropped.