5 ms·
This, taken in combination with the SAE paper, the golden-gate claude paper, the feelings / introspection paper, and note in the fable system card (that they ar
by minimaltom 3mo ago
This, taken in combination with the SAE paper, the golden-gate claude paper, the feelings / introspection paper, and note in the fable system card (that they are silently nerfing responses about activation shaping), is basically confirmation to me that they have a new technique they they are using during training (along the vibe space of these mechinterp papers), and its probably some kind of representation learning akin to the core ideas of JEPA.
(Nb: not an expert / in the labs, just opining)
- Smaug123 3mo agoNote that Neel Nanda replicated the results on a Qwen model.
- aesthesia 3mo agoAnd Neuronpedia released Jacobian lens weights for a wide range of open models: https://huggingface.co/neuronpedia/jacobian-lens https://huggingface.co/neuronpedia/jacobian-lens
- orbital-decay 3mo agoI'm sure Anthropic of all companies don't do that, since using mechinterp as a training target will make the the result uninterpretable.
- mike_hearn 3mo agoThey're definitely willing to consider it. Read the parts of the paper where they use J-space interpretations to train more ethical behavior into the model by interrupting it.