3 ms·

This result feels very intuitive. The early layers of a transformer can be thought of as understanding surface level things like syntax, how tokens group, which

This result feels very intuitive. The early layers of a transformer can be thought of as understanding surface level things like syntax, how tokens group, which groups are entities and how to disambiguate them, etc. The last layers are in a sense decoding ideas into a selection of words, ensuring the grammar makes sense, that the text flows and is structured correctly, etc. The middle layers are where the abstract thought and manipulation of concepts is happening. But for the tasks this paper uses for RL training, it's all about improving the way the net is manipulating concepts. So the middle layers are where the focus should be. Note: RL is also used for tasks that aren't about conceptual manipulation, like instruct training. I bet that their result doesn't hold for that because the delta vs the foundation model is all about the selection of words and flow of the text, not the core understanding.

throw310822 4mo ago
I keep thinking of the RYS (Repeat Yourself) experiment of simply looping some of the inner layers of LLMs for better results and wonder if any progress was made on it. https://dnhkng.github.io/posts/rys/ https://dnhkng.github.io/posts/rys/ Feels it should be straightforward to integrate in LLMs a network to control the looping. Or just duplicate entire blocks of layers after the initial training.
- naasking 4mo ago
  Yes, computing in latent space is a big thing now. https://ouro-llm.github.io/ https://ouro-llm.github.io/
- kridsdale1 4mo ago
  Is that in effect doing the same as “chain of thought” or “think through your steps” aka “Reasoning Models”?
  - 8note 4mo ago
    there certainly are experiments in keeping the reasoning in latent space. i dont think this is quite the same though, since you arent picking tokens for the chain of thought. inatead, its staying on trying to pick the immediate next token. as an alternative, maybe you could stack these to produce most likely token lists instead by stacking these? but i think youd end up with the similar blurriness that llm video generators get where theyre returning an average of all the likely combinations rather than collapsing that wave function
  - mike_hearn 4mo ago
    It seems related. The primary benefit of reasoning tokens seems to be just giving the model more chances to loop through all its layers. There were experiments at one point with "pause tokens" that were meaningless but seemed to improve performance. Reasoning is a lot more interpretable though.
idiotsecant 4mo ago
I've often thought about how rich a machine language could be that communicates machine to machine on an interface that is really really close to those middle layers. I imagine a standardize meta interface that each model 'grows' a connection to with RL.
- mike_hearn 4mo ago
  This idea is called neuralese and the labs are reluctant to do it because of the interpretability and control issues it poses.

throw310822 4mo ago

I keep thinking of the RYS (Repeat Yourself) experiment of simply looping some of the inner layers of LLMs for better results and wonder if any progress was made on it. https://dnhkng.github.io/posts/rys/ https://dnhkng.github.io/posts/rys/ Feels it should be straightforward to integrate in LLMs a network to control the looping. Or just duplicate entire blocks of layers after the initial training.

naasking 4mo ago
Yes, computing in latent space is a big thing now. https://ouro-llm.github.io/ https://ouro-llm.github.io/
kridsdale1 4mo ago
Is that in effect doing the same as “chain of thought” or “think through your steps” aka “Reasoning Models”?
- 8note 4mo ago
  there certainly are experiments in keeping the reasoning in latent space. i dont think this is quite the same though, since you arent picking tokens for the chain of thought. inatead, its staying on trying to pick the immediate next token. as an alternative, maybe you could stack these to produce most likely token lists instead by stacking these? but i think youd end up with the similar blurriness that llm video generators get where theyre returning an average of all the likely combinations rather than collapsing that wave function
- mike_hearn 4mo ago
  It seems related. The primary benefit of reasoning tokens seems to be just giving the model more chances to loop through all its layers. There were experiments at one point with "pause tokens" that were meaningless but seemed to improve performance. Reasoning is a lot more interpretable though.

idiotsecant 4mo ago

I've often thought about how rich a machine language could be that communicates machine to machine on an interface that is really really close to those middle layers. I imagine a standardize meta interface that each model 'grows' a connection to with RL.

mike_hearn 4mo ago
This idea is called neuralese and the labs are reluctant to do it because of the interpretability and control issues it poses.