3 ms·
Transformers scale in training mainly because they are stateless! They have a long lookback window (30,000 tokens in GPT4!) so the issues are smoothed over. L
by VHRanger 4y ago
Transformers scale in training mainly because they are stateless!
They have a long lookback window (30,000 tokens in GPT4!) so the issues are smoothed over.
LSTMs end up being very finnicky to train and their sequential nature makes model scaling bottlenecked in parallelism.
- gliptic 4y agoRWKV training can be parallelized just like transformers. It's not LSTM.
- VHRanger 4y agoRight, I meant why transformers exploded over LSTM-based LMs in 2018 was because of those training issues that transformers solved. I'm excited about the next iteration of models focusing on inference. RWVK seems promising on this end. Also love the citizen science aspect of RWVK coming out.