3 ms·
This deepmind paper showed big limitations on nested structures with transformers, LSTMs actually did better: Neural Networks and the Chomsky Hierarchy https:
by cma 3mo ago
This deepmind paper showed big limitations on nested structures with transformers, LSTMs actually did better:
Neural Networks and the Chomsky Hierarchy
https://arxiv.org/abs/2207.02098 https://arxiv.org/abs/2207.02098
- galaxyLogic 3mo agoInteresting. Can someone quickly summarize their results? Are they saying that using languages with simpler syntax Lisp lead to LLM being able to generalize their knowledge better?
- cma 3mo agoThey tested them on formal languages of different power, and saw where they could generalize beyond the training data. Transformers failed to generalize pretty early on at stack machines/brace matching. It was a good bit older of a paper though, if I remember it's somewhat expected from the pure feed forward nature of them and limited circuit depth, where LSTMs have some recurrence.