Deepmind did a study of which language classes different neural network architectures can learn with generalization: https://arxiv.org/abs/2207.02098 https://ar

Deepmind did a study of which language classes different neural network architectures can learn with generalization: https://arxiv.org/abs/2207.02098 https://arxiv.org/abs/2207.02098 ("Neural Networks and the Chomsky Hierarchy") LSTMs could learn slightly deeper into the Chomsky hierarchy than transformers, neural turing machines could learn them all including recursively enumerable (context free grammar or turing machine to produce). Transformers did outperform LSTMs in something in an alternate hierarchy if I remember. It's not conclusive since it is not training indefinitely, there are formal proofs that many architectures can do things that they can't actually effectively be trained for, but they threw a lot of compute at it.