Noob here. Can one make the following deduction: In transformer based architectures, where one typically uses variation of attention mechanism to model interac

Noob here. Can one make the following deduction: In transformer based architectures, where one typically uses variation of attention mechanism to model interactions, even if one does not consider the autoregressive assumption of the domain's "nodes"(amino acids, words, image patches), if the number of final states that nodes take eventually can be permuted only in a finite way(i.e. they have sparse interactions between them), then these architectures are efficient way of modeling such domains. In plain english the final state of words in a sentence and amino acids in a protein have only so many ways they can be arranged and transformers do a good job of modeling it. Also can one assume this won't do well for domains where there is, say, sensitivity to initial conditions, like chaotic systems like wheather where the # final states just explodes?