3 ms·
I keep thinking of the RYS (Repeat Yourself) experiment of simply looping some of the inner layers of LLMs for better results and wonder if any progress was mad
by throw310822 3mo ago
I keep thinking of the RYS (Repeat Yourself) experiment of simply looping some of the inner layers of LLMs for better results and wonder if any progress was made on it.
https://dnhkng.github.io/posts/rys/ https://dnhkng.github.io/posts/rys/
Feels it should be straightforward to integrate in LLMs a network to control the looping. Or just duplicate entire blocks of layers after the initial training.
- naasking 3mo agoYes, computing in latent space is a big thing now. https://ouro-llm.github.io/ https://ouro-llm.github.io/
- kridsdale1 3mo agoIs that in effect doing the same as “chain of thought” or “think through your steps” aka “Reasoning Models”?
- 8note 3mo agothere certainly are experiments in keeping the reasoning in latent space. i dont think this is quite the same though, since you arent picking tokens for the chain of thought. inatead, its staying on trying to pick the immediate next token. as an alternative, maybe you could stack these to produce most likely token lists instead by stacking these? but i think youd end up with the similar blurriness that llm video generators get where theyre returning an average of all the likely combinations rather than collapsing that wave function
- mike_hearn 3mo agoIt seems related. The primary benefit of reasoning tokens seems to be just giving the model more chances to loop through all its layers. There were experiments at one point with "pause tokens" that were meaningless but seemed to improve performance. Reasoning is a lot more interpretable though.