The distinctive aspect of the 'transformer' family of neural networks is that they incorporate 'attention', which is a model that identifies which parts of the

The distinctive aspect of the 'transformer' family of neural networks is that they incorporate 'attention', which is a model that identifies which parts of the input are critical to its meaning. You're correct in that modern transformer models are essentially Markov chains, but the function that derives the probabilities (of which the attention heads are part of) is imbued with 'understanding' of related concepts during the training process. In contrast, a traditional Markov chain text generator might have a probability function that only takes into account the frequency of the n-grams in the training data, and so produces superficially coherent (but mostly meaningless) output.