3 ms·
Technically, transformers condition on the entire past not just the last step, but RNNs are Markov Chains. RNNs have information bottleneck issues though.
by visarga 3y ago
Technically, transformers condition on the entire past not just the last step, but RNNs are Markov Chains. RNNs have information bottleneck issues though.
- uoaei 3y agoTransformers (for NLP) also perform steps on Markov chains. The difference is that with transformers (for NLP), which Markov chain it's moving along changes every step.