4 ms·
Have any of the "sub-quadratic transformers" [1] gone mainstream? Or is everyone simply rich enough to buy enough GPUs. [1]https://www.gwern.net/notes/Attentio
by algo_trader 5y ago
Have any of the "sub-quadratic transformers" [1] gone mainstream? Or is everyone simply rich enough to buy enough GPUs.
[1]https://www.gwern.net/notes/Attention https://www.gwern.net/notes/Attention
- lucidrains 5y agoI would recommend Routing Transformer https://github.com/lucidrains/routing-transformer https://github.com/lucidrains/routing-transformer but the real truth is nothing beats full attention. Luckily, someone recently figured out how to get past the memory bottleneck. https://github.com/lucidrains/memory-efficient-attention-pytorch https://github.com/lucidrains/memory-efficient-attention-pyt...