3 ms·
I'm surprised more attention isn't paid to this research direction, that nobody has tried to generalize it for example by combining the recurrence concept with
by mysterEFrank 11mo ago
I'm surprised more attention isn't paid to this research direction, that nobody has tried to generalize it for example by combining the recurrence concept with next token prediction.
That said despite the considerable gains this seems to just be some hyperparameter tweaking rather than a foundational improvement.
- whiplash451 11mo agoNot just hyper parameter tweaking. Not foundational research either. But rather engineering improvements that compound with each other (conswiglu layers, muon optimizer)
- in-silico 11mo ago> nobody has tried to generalize it for example by combining the recurrence concept with next token prediction Here you go: https://arxiv.org/abs/2502.05171 https://arxiv.org/abs/2502.05171
- mysterEFrank 11mo agoThanks! This seems to work incredibly well.