4 ms·
Neural Networks and the Chomsky Hierarchy by DeepMind
- ashvardanian 4y agoDCF Reverse String task is my personal favorite, when evaluating sequnece-processing neural nets. So simple, yet puzzles systems with billions of parameters. Love the conclusion too: > In particular, we showed that state-of-the-art architectures, such as LSTMs and Transformers, cannot solve seemingly simple tasks, such as duplicating a string, when evaluated on sequences that are significantly longer than those seen during training. Moreover, we showed that models interacting with an external memory structure, such as a stack or a finite tape, can climb the Chomsky hierarchy, indicating a promising direction for improvements in architecture design. Which other labs work in this direction?