3 ms·
I love reading about papers like these. They raise hopes that novel model architectures might reduce the need for computational resources to train powerful mode
by samus 3y ago
I love reading about papers like these. They raise hopes that novel model architectures might reduce the need for computational resources to train powerful models, which would help lower the barrier of entry.
- heyitsguay 3y agoIf you read "Attention Is All You Need", you'll see that was actually the motive behind the original Transformer too! Bit of a Jevon's paradox situation.
- zozbot234 3y agoTransformer was really about making it feasible to parallelize training of large context windows. More like using large compute resources effectively when available than saving on compute. This is also why it looks quite ad-hoc compared to simpler models like RNN or LSTM, which can be trained in large batches of data (hence still making use of parallelism to some extent) but have to be serial along the context dimension.