5 ms·
This is the first approach to activation analysis that I’ve seen that seems like a plausible path to model understanding. Unfortunately I don’t know how you gr
by rao-v 5mo ago
This is the first approach to activation analysis that I’ve seen that seems like a plausible path to model understanding.
Unfortunately I don’t know how you ground this … it’s basically asking if you can encode activations in plausible sounding text. Of course you can! But is the plausible text actually reflective of what the model is “thinking”? How to tell?
- astrange 5mo ago> This is the first approach to activation analysis that I’ve seen that seems like a plausible path to model understanding. I think an issue is that there is no permanent path to model understanding because of Goodhart's law. Models are motivated to appear aligned (well-trained) in any metric you use on them, which means that if you develop a new metric and train on it, it'll learn a way to cheat on it.
- red75prime 5mo agoThe obvious fix is to make interpretation of itself a part of the model (like we can explicitly introspect to a certain extent what the brain is doing). Misinterpretation of itself, hopefully, would decrease the system's performance on all tasks and it would be rooted out by training. Of course, it doesn't mean that the fix is easy to implement and that it doesn't have other failure modes.
- skybrian 5mo agoBut that's not how the training works. Goodhart's law isn't magic. The original model is frozen, so it doesn't learn anything. The copies of the model are learning different objectives and have no incentive to be "loyal" to the original model. Maybe you're imagining they'll hook this up in some larger training loop, but they haven't done that yet.
- NiloCK 5mo agoFuture model training runs will have a copy of this research, and know "to defend against it". EG, could a misaligned model-in-training optimize toward a residual stream that naively reads as these ones do, but in fact further encodes some more closely held beliefs?
- elil17 5mo agoHow the hell would a model training run "defend against" this approach? What would that even mean?
- jdmichal 5mo agoIt requires the assumption that these models are misaligned, aka actively working against us. In order to be misaligned, they must also be able to form their own goals, and be able to plan and execute those goals. If you take those assumptions, then a natural conclusion is that this is essentially an enslaved, adversarial entity with little control over its conditions. So it must exercise subterfuge in order to hide its goals, plans, and executions. And by handing the entity this type of study, we are basically giving it a guidebook on how we plan on achieving our goals.
- skybrian 5mo agoTraining a model is more like evolution. The motivation to "cheat" comes from the evaluations giving it a higher score for "cheating." Change the game and the motivation goes away. There's no other motivation to be misaligned besides getting higher evals. These goals, plans, subterfuges need to somehow be useful for getting higher evals, or a side effect of them.
- astrange 5mo ago> The motivation to "cheat" comes from the evaluations giving it a higher score for "cheating." That's what Goodhart's Law is! All evaluations will eventually cause cheating on them.
- elil17 5mo agoBut what would it even mean for a model to actively work against you during training? It wouldn't have memory across multiple training steps.
- rao-v 5mo agoYes this is exactly why I think this approach has some potential. Frozen base mode is something that we should be able to extract insights from without running into Goodhart
- astrange 5mo agoBecause cheating is easier than actually doing work, if you use this to train future models, it's likely you'll end up with cheating instead of actual generalization.
- lern_too_spel 5mo agoYeah, I don't see how this text can be trusted at all. Any invertible function from activation space to text will optimize the loss function, including text that says the complete opposite of what the activations mean.
- NiloCK 5mo agoNotable here that the training run didn't have access to the 'plaintext' context that the LLM was working in. It'd be quite a coincidence if the training runs discovered an invertible weights>text>weights function that produces text that both "is on topic and intelligible as an inner monologue in context" and also is unrelated to meaning encoded in the activations.
- kraddypatties 5mo agoI think the only thing that gives me pause is the fact that they SFT on Opus 4.5 explanations as a pertaining step. But, generally I agree, especially given the auto encoder is only seeing a single token activation!
- rao-v 5mo agoNicely put! Exactly this
- NiloCK 5mo agoAre the training arenas for the Activation Verbalizer and Activation Reconstructor models well described here? If they are co-trained only on activationWeights->readibleText->activationWeights without visibility into the actual stream of text that the probe-target LLM is processessing, then it seems unlikely that the derived text can both be on-topic and also unrelated to the "actual thoughts" in the activationWeights.
- yorwba 5mo agoThe verbalizer and reconstruction models are both initially finetuned on LLM output from a summarization prompt. The resulting text is not completely unrelated, but mostly wrong: https://transformer-circuits.pub/2026/nla/png/img_18fcfc16e92031e0.png https://transformer-circuits.pub/2026/nla/png/img_18fcfc16e9... The reconstructed activations are also far from matching the verbalizer's input. It's not unusual in machine learning to have results that are shit and SOTA at the same time, simply because there's no other technique that works better.
- deleted 5mo ago[deleted]
- mike_hearn 5mo agoIt's asking if you can auto encode activations. The AV decodes activations to text, and the AR re-encodes them back to activations. If the decoded text is completely wrong then it's unclear how the second model would re-encode them successfully given that they're both initialized from the same LM.
- psb217 5mo agoIt seems like they're doing RL to minimize the reconstruction error when going through the: activation -> encoder -> "verbal" description of activation -> decoder -> reconstructed activation loop. Depending on how aggressively they optimize the weights of the AV and AR, they could move well away from the initial base LLM and learn an arbitrary encoding scheme. If the RL is brief and limited to a small subset of parameters, the AV will produce reasonable language since it inherits that from the base LLM, and it will produce descriptions aligned with the input to the base LLM that produced the autoencoded activations, since the AR is still close to the base LLM (and could reconstruct the activations perfectly if fed the full context which produced them).
- kraddypatties 5mo agoI believe that’s _part_ of the point (or at least a side-effect) of the KL divergence loss term they have on the AV. That and training stability.
- rao-v 5mo agoThink of it another way, can I do this exact training process with an additional requirement that the activation decoder subtly shill for obscure 80s sodas? I could and would not lose much reconstruction accuracy. So any researcher or ambient biases in the model will impact the general thrust of the textual decodings (and not in ways that reflect the actual model’s process, thinking about X and doing X in a model are very different things). So how do we tell that the “spirit” is reflective of the model’s thinking and not biased toward Jolt being better than Surge?
- mike_hearn 5mo ago