2 ms·
It helps to start with recurrent neural networks first, since those were the previous standard way of doing next-token-prediction. They worked, but training the
by Analemma_ 3y ago
It helps to start with recurrent neural networks first, since those were the previous standard way of doing next-token-prediction. They worked, but training them was extremely slow because it couldn't be parallelized. Transformers are a way of getting mostly the same capabilities as RNNs but with a parallelizable architecture so you can actually train it with huge parameter numbers in a reasonable amount of time.
- microtonal 3y agoI agree it's very useful to look at RNNs. Even more because attention mechanisms were already getting popular in LSTM/GRU models before the transformers paper. One of the main insights of the transformer papers is that you can do with just an attention attention and you do not need the recurrent part of the network anymore (hence the title "Attention is all you need").