3 ms·
If you read "Attention Is All You Need", you'll see that was actually the motive behind the original Transformer too! Bit of a Jevon's paradox situation.
by heyitsguay 3y ago
If you read "Attention Is All You Need", you'll see that was actually the motive behind the original Transformer too! Bit of a Jevon's paradox situation.
- zozbot234 3y agoTransformer was really about making it feasible to parallelize training of large context windows. More like using large compute resources effectively when available than saving on compute. This is also why it looks quite ad-hoc compared to simpler models like RNN or LSTM, which can be trained in large batches of data (hence still making use of parallelism to some extent) but have to be serial along the context dimension.