4 ms·
Yeah, I don't see how this text can be trusted at all. Any invertible function from activation space to text will optimize the loss function, including text tha
by lern_too_spel 6mo ago
Yeah, I don't see how this text can be trusted at all. Any invertible function from activation space to text will optimize the loss function, including text that says the complete opposite of what the activations mean.
- NiloCK 6mo agoNotable here that the training run didn't have access to the 'plaintext' context that the LLM was working in. It'd be quite a coincidence if the training runs discovered an invertible weights>text>weights function that produces text that both "is on topic and intelligible as an inner monologue in context" and also is unrelated to meaning encoded in the activations.
- kraddypatties 6mo agoI think the only thing that gives me pause is the fact that they SFT on Opus 4.5 explanations as a pertaining step. But, generally I agree, especially given the auto encoder is only seeing a single token activation!
- rao-v 6mo agoNicely put! Exactly this