3 ms·

So, does "reasoning in latent space" mean that it does/will not use tokens (e.g. using CoT etc.)?

• Sharlin 2mo ago
  The computation still has to be done, even if it's more efficient in a higher-entropy space.
• zarzavat 2mo ago
  Models already reason in latent space, but they have to keep encoding-decoding their "thoughts" from/to human interpretable tokens, and notably those tokens are sampled from a distribution. The model cannot output a vector and have that same vector fed back in at the next step, it only sees what token the sampler collapsed its vector into. It's as if the only way you could think was by writing down a word, erasing all the thoughts from your head, then reading the word you just wrote down and deciding on the next word, etc. Reasoning purely in latent space means that the model would still produce an output equivalent to tokens but unconstrained e.g. the output could be raw and opaque vectors. A significant downside is that you lose the ability to inspect the reasoning trace. It would also make the reasoning trace potentially larger which has operational issues.
  • jacquesm 2mo ago
    More like erasing all other branches just prior to that word from your head, the past thoughts would still remain.
  • 0-_-0 2mo ago
    The kv cache is not discretised, only the output tokens.
  • wren6991 2mo ago
    > The model cannot output a vector and have that same vector fed back in at the next step, it only sees what token the sampler collapsed its vector into. Not completely true: KV is a projection of the activation at each layer's input, so attention heads see (a representation of) all previous tokens' activations at that layer. The hard decision at the LM head doesn't change that.