3 ms·
> Industry OTOH has gone all-in on Transformers. It's so annoying. Transformers keep improving and recurrent networks are harder to train so until we hit some
by viraptor 9mo ago
> Industry OTOH has gone all-in on Transformers.
It's so annoying. Transformers keep improving and recurrent networks are harder to train so until we hit some real wall, companies don't seem eager to diverge. It's like lithium batteries improving easy faster than it was profitable to work on sodium ones, even though we unfortunately want the sodium ones to be better.
- cs702 9mo agoI'd add this to the list of linear-attention RNNs: https://arxiv.org/abs/2602.00294 https://arxiv.org/abs/2602.00294 Recently saw it on HN.