You don't think that complex structures are required to generate well formed text? Human language is extremely context dependent, and dealing with that is what

You don't think that complex structures are required to generate well formed text? Human language is extremely context dependent, and dealing with that is what all those macro structures does. Context dependence is literally the novel thing with transformers, they are context dependent statistical models to generate next words, make it huge and feed it a page of context and it can map those to a page of output to match that context.