3 ms·
If you read the rest of the paper it's clear what they mean. SSMs are theoretically no more powerful than transformers, meaning without chain of thought they ca
by logicchains 2y ago
If you read the rest of the paper it's clear what they mean. SSMs are theoretically no more powerful than transformers, meaning without chain of thought they can't efficiently solve state-tracking problems, unlike RNNs: https://arxiv.org/abs/2207.00729 https://arxiv.org/abs/2207.00729 .
- opprobium 2y agoNot just efficiently, can't solve.
- logicchains 2y agoThey can solve it if you keep adding layers to the transformer, it's just not efficient; you'd need exponentially more layers than a similarly sized RNN.
- yobbo 2y agoThe difference to RNNs is that the state in SSMs is a linear combination of previous inputs, which makes it possible to parallelise training in various ways. Gates and non-linearities in RNNs allow the state to be "any" function of previous inputs.