3 ms·

Is that in effect doing the same as “chain of thought” or “think through your steps” aka “Reasoning Models”?

• 8note 4mo ago
  there certainly are experiments in keeping the reasoning in latent space. i dont think this is quite the same though, since you arent picking tokens for the chain of thought. inatead, its staying on trying to pick the immediate next token. as an alternative, maybe you could stack these to produce most likely token lists instead by stacking these? but i think youd end up with the similar blurriness that llm video generators get where theyre returning an average of all the likely combinations rather than collapsing that wave function
• mike_hearn 4mo ago
  It seems related. The primary benefit of reasoning tokens seems to be just giving the model more chances to loop through all its layers. There were experiments at one point with "pause tokens" that were meaningless but seemed to improve performance. Reasoning is a lot more interpretable though.