4 ms·
I'm surprised more attention isn't paid to this research direction, that nobody has tried to generalize it for example by combining the recurrence concept with
by mysterEFrank 10mo ago
I'm surprised more attention isn't paid to this research direction, that nobody has tried to generalize it for example by combining the recurrence concept with next token prediction.
That said despite the considerable gains this seems to just be some hyperparameter tweaking rather than a foundational improvement.
- whiplash451 10mo agoNot just hyper parameter tweaking. Not foundational research either. But rather engineering improvements that compound with each other (conswiglu layers, muon optimizer)
- in-silico 10mo ago> nobody has tried to generalize it for example by combining the recurrence concept with next token prediction Here you go: https://arxiv.org/abs/2502.05171 https://arxiv.org/abs/2502.05171
- mysterEFrank 10mo agoThanks! This seems to work incredibly well.