3 ms·
> New LLMs don't even rely that much on that aforementioned older architecture Don’t they all indicate being based on the transformer architecture? > not enti
by espadrine 2y ago
> New LLMs don't even rely that much on that aforementioned older architecture
Don’t they all indicate being based on the transformer architecture?
> not entirely because of transformers but because of the hardware
Kaplan et al. 2020[0] (figure 7, §3.2.1) shows that LSTMs, the leading language architecture prior to transformers, scaled worse because they plateau’ed quickly with larger context.
[0]: https://arxiv.org/abs/2001.08361 https://arxiv.org/abs/2001.08361