14 ms·
Unfortunately I think the competitive pressures to improve model performance makes this kind of monitor-ability is short-lived. It's just not likely that textua
by vonneumannstan 1y ago
Unfortunately I think the competitive pressures to improve model performance makes this kind of monitor-ability is short-lived. It's just not likely that textual reasoning in english ismost optimal.
Researchers are already pushing in this direction:
https://arxiv.org/abs/2502.05171 https://arxiv.org/abs/2502.05171
"We study a novel language model architecture that is capable of scaling test-time computation by implicitly reasoning in latent space. Our model works by iterating a recurrent block, thereby unrolling to arbitrary depth at test-time. This stands in contrast to mainstream reasoning models that scale up compute by producing more tokens. Unlike approaches based on chain-of-thought, our approach does not require any specialized training data, can work with small context windows, and can capture types of reasoning that are not easily represented in words. We scale a proof-of-concept model to 3.5 billion parameters and 800 billion tokens. We show that the resulting model can improve its performance on reasoning benchmarks, sometimes dramatically, up to a computation load equivalent to 50 billion parameters."
https://arxiv.org/abs/2412.06769 https://arxiv.org/abs/2412.06769
Can't monitor the chain of thought if it's no longer in a human legible format.
- dlivingston 1y agoWould it be possible to train a language translation model to map from latent space -> English? If my understanding is correct, a plain-text token is just a point in latent space mapped to an embedding vector. Any reasoning done in latent space is therefore human-readable as a sequence of raw tokens. I'm not sure what the token sequence would look like at this point -- I assume they're full or partial (mainly English) words, connected together by the abstract N-dimensional latent space concept of the token, not connected grammatically. Something like: > prompt: add 2 + 2 > reasoning: respond computation mathematics algebra scalar integer lhs 2 rhs 2 op summation > lhs 2 rhs 2 op summation gives 4 > computation ops remain none result 4 > response: 4 Something like that; probably even less sensical. Regardless, that could be "language translated" to English easily. I have not read the paper so this may have been addressed.
- sailingparrot 1y ago> Any reasoning done in latent space is therefore human-readable as a sequence of raw tokens. The mapping from latent space to logits (a score for how suitable the model think each possible token is to be the next one in the sentence) is one of the most costly operation of the entire LLM, so when doing reasoning in latent space you typically want to introduce some kind of recurrence somewhere before this mapping takes place, and only go through it once the model is "done" reasoning, at which point it becomes human interpretable. > would it be possible to train a language translation model to map from latent space -> English It's hard and expensive and slow but some progress is being made here: https://transformer-circuits.pub/2025/attribution-graphs/biology.html https://transformer-circuits.pub/2025/attribution-graphs/bio...
- vonneumannstan 1y agoI think the larger issue is eventually the models might 'think' by passing massive matrices of floating point numbers around and there's basically no hope of decoding those.
- AstralStorm 1y agoIt might even be legible, just encoded in such a way that the monitor would only flag it as unrecognized nonsense of a hallucination and be unable to detect purposeful reasoning inside it.