3 ms·
People are trying to solve it with software too, even if you don't hear about it. The most high-profile example is the latest set of Qwen models, which replace
by in-silico 6mo ago
People are trying to solve it with software too, even if you don't hear about it.
The most high-profile example is the latest set of Qwen models, which replace most of the attention mechanisms with Gated DeltaNet (which uses constant memory with respect to sequenc length).
Test-time training architectures are also getting a lot of attention, and have shown great performance in the acedemic setting. It's only a matter of time before we start getting open TTT models.