9 ms·
Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
- HanClinto 2y agoReminds me of this paper from a couple of weeks ago that isolated the "refusal vector" for prompts that caused the model to decline to answer certain prompts: https://news.ycombinator.com/item?id=40242939 https://news.ycombinator.com/item?id=40242939 I love seeing the work here -- especially the way that they identified a vector specifically for bad code. I've been trying to explore the way that we can use adversarial training to increase the quality of code generated by our LLMs, and so using this technique to get countering examples of secure vs. insecure code (to bootstrap the training process) is really exciting. Overall, fascinating stuff!!
- watersb 2y agoAm I the only one to read 'monosemanticity' as 'moose-mantically'? Like, its talking about moose magick...
- astrange 2y ago> Many features are multilingual (responding to the same concept across languages) and multimodal (responding to the same concept in both text and images), as well as encompassing both abstract and concrete instantiations of the same idea (such as code with security vulnerabilities, and abstract discussion of security vulnerabilities). This seems like it's trivially true; if you find two different features for a concept in two different languages, just combine them and now you have a "multilingual feature". Or are all of these features the same "size"? They might be and I might've missed it.
- wrycoder 2y agoThey are trying to figure out what they actually built. I suspect the time is coming when there will always be an aligned search AI between you and the internet.
- deleted 2y ago[deleted]
- justanotherjoe 2y agoMy thoughts - LLM Just got a whole set of buttons you can push. Potential for the LLM to push its own buttons? - Read the paper and ctrl+f 'deplorable'. This shows once again how we are underestimating LLM's ability to appear conscious. It can be really effective. Reminiscence of Dr.Ford in Westworld :'you (robots) never look more human than when you are suffering.' Or something like that, anyway. I might be hallucinating dialogue but pretty sure something like that was said and I think it's quite true. - Intensely realistic roleplaying potential unlocked. - Efficiency by reducing context length by directly amplifying certain features instead. Very powerful stuff. I am waiting eagerly when I can play with it myself. (Someone please make it a local feature)
- byteknight 2y agoThis reminds me of how people often communicate to avoid offending others. We tend to soften our opinions or suggestions with phrases like "What if you looked at it this way?" or "You know what I'd do in those situations." By doing this, we subtly dilute the exact emotion or truth we're trying to convey. If we modify our words enough, we might end up with a statement that's completely untruthful. This is similar to how AI models might behave when manipulated to emphasize certain features, leading to responses that are not entirely genuine.
- nathan_compton 2y agoCounterpoint: "What if you looked at it this way?" communicates both your suggestion AND your sensitivity to the person's social status whatever. Given that humans are not robots, but social, psychological, animals, such communication is entirely justified and efficient.
- byteknight 2y agoYou can't always do both to the fullest truth. They often conflict. To do what you suggest, would imply my feelings perfectly align with the sympathetic view. That is not the case for a lot of humans or instances. If I am not saying exactly how I feel it is watered down. And telling me "just do both" is enforcing your world view and that is precisely what we're talking about _not_ doing.
- infogulch 2y agoThe "fullest truth" includes your desired outcome and knowledge that they are a human. If you just want to dump facts at them and get them to shut up, go ahead and speak unfiltered. Twitter may be an example of the outcome of that strategy. Consider a situation where you are teaching a child. She tries her best and makes a mistake on her math homework. Saying that her attempt was terrible because an adult could do better may be the "fullest truth" in the most eye-rolling banal way possible, and discourages her from trying in the future which is ultimately unproductive. This "fullest truth" argument fails to take into account desire and motivation, and thus is a bad model of the truth.
- optimalsolver 2y ago>what the model is "thinking" before writing its response An actual "thinking machine" would be constantly running computations on its accumulated experience in order to improve its future output and/or further compress its sensory history. An LLM is doing exactly nothing while waiting for the next prompt.
- fassssst 2y agoWhy does the timing of the “thinking” matter?
- verdverm 2y agothinking is generally considered an internal process, without input/output (of tokens), though some people decide to output some of that thinking into a more permanent form I see thinking as less about "timing" and more about a "process" What this post seems to be describing is more about where attention is paid and what neurons fire for various stimuli
- sabrina_ramonov 2y agowe know so little about thinking and consciousness, these claims seem premature
- verdverm 2y agoThat one can fix the RNG and get consistent output indicates a lack of dynamics They certainly do not self update the weights in an online process as needed information is experienced
- whimsicalism 2y agoIf we could perfectly simulate the brain and there were quantum hidden variables, we too could “fix RNG and get deterministic output”
- 2y ago
- feverzsj 2y agoSo they made a system by trying out thousands of combinations to find the one gives best result, but they don't understand what's actually going on inside.
- whimsicalism 2y agoI continue to be impressed by Anthropic’s work and their dual commitment to scaling and safety. HN is often characterized by a very negative tone related to any of these developments, but I really do feel that Anthropic is trying to do a “race to the top” in terms of alignment, though it doesn’t seem like all the other major companies are doing enough to race with them. Particularly frustrating on HN is the common syllogism of: 1. I believe anything that “thinks” must do X thing. 2. LLM doesn’t do X thing 3. LLM doesn’t think X thing is usually both poorly justified as constitutive of thinking (usually constitutive of human thinking but not writ large) nor is it explained why it matters whether the label of “thinking” applies to LLM or not if the capabilities remain the same.
- handwarmers 2y agoWhat is often frustrating to me at least is the arbitrary definition of "safety" and "ethics", forged by a small group of seemingly intellectually homogenous individuals.
- whimsicalism 2y agoSay more, this is half a thought
- CamperBob2 2y agoE.g., the common sentiment that "NSFW" output is to be prohibited, regardless of whether you work in a steel mill or a church.
- ben_w 2y agoYes, even though this is a mild improvement on 20 years ago when it was an even more homogenous group. Given how often China comes up in the context of AI, I'm wondering: Lots of people in the West treat China as mysterious and alien. I wonder how true that really is (e.g. Confucianism)? Or if it ever was (e.g. perhaps it used to be before industrialisation, which homogenises everyone regardless of the origin)?
- 2y ago
- null_point 2y agoStrategic timing for the release of this paper. As of last week OpenAI looks weak in their commitment to _AI Safety_, losing key members of their Super Alignment team.
- tel 2y agoThis is exceptionally cool. Not only is it very interesting to see how this can be used to better understand and shape LLM behavior, I can’t help but also think it’s an interesting roadmap to human anthropology. If we see LLMs as substantial compressed representations of human knowledge/thought/speech/expression—and within that, a representation of the world around us—then dictionary concepts that meaningfully explain this compressed representation should also share structure with human experience. I don’t mean to take this canonically, it’s representations all the way down, but I can’t help but wonder what the geometry of this dictionary concept space says about us.
- davedx 2y agoThe vector space projection of the human experience. I like it.
- wwarner 2y agohuge. the activation scan, which looks for which nodes change the most when prompted with the words "Golden Gate Bridge" and later an image of the same bridge, is eerily reminiscent of a brain scan under similar prompts...
- verdverm 2y agoI find this outcome expected and not really surprising, more confirmation of previous results. Consider vision transformers and the papers that showed what each layer was focused on.
- wwarner 2y agowell that's exactly the point -- no such result is available for language models.
- verdverm 2y agoThere are multiple papers and efforts that have inspected the internal state of LLMs. One could even see the word2vec analysis along these lines, as evidence that the model is specializing neurons One such example: The Internal State of an LLM Knows When It's Lying (https://arxiv.org/abs/2304.13734 https://arxiv.org/abs/2304.13734) Searching phrases like "llm interpretability" and "llm activation analysis" uncover more https://github.com/JShollaj/awesome-llm-interpretability https://github.com/JShollaj/awesome-llm-interpretability
- wwarner 2y agoYes, lots of activity in the space. I thought you were saying it was a dumb problem, but I was wrong. I think this is a great paper.
- verdverm 2y agoyup, if you look at drop out, what it does and why, you can see additional interesting results along these lines (drop-out was found to increase resilience in models because they had to encode information in the weights differently, i.e. could not rely on single neuron (at the limit))
- pdevr 2y agoSo, to summarize: >Used "dictionary learning" >Found abstract features >Found similar/close features using distance >Tried amplifying and suppressing features Not trying to be snary, but sounds mundane in the ML/LLM world. Then again, significant advances have come from simple concepts. Would love to hear from someone who has been able to try this out.
- sjkoelle 2y agothe interesting advance in the anthropic/mats research program is the application of dictionary learning to the "superpositioned" latent representations of transformers to find more "interpretable" features. however, "interpretability" is generally scored by the explainer/interpreter paradigm which is a bit ad hoc, and true automated circuit discovery (rather than simple concept representation) is still a bit off afaik.
- youssefabdelm 2y agoFor anyone who has read the paper, have they provided code examples or enough detail to recreate this with, say, Llama 3? While they're concerned with safety, I'm much more interested in this as a tool for controllability. Maybe we can finally get rid of the woke customer service tone, and get AI to be more eclectic and informative, and less watered down in its responses.
- tantalor 2y agoI always assumed the way to map these models would be by ablation, the same way we map the animal brain. Damage part X of the network and see what happens. If the subject loses the ability to do Y, then X is responsible for Y. See https://en.wikipedia.org/wiki/Phineas_Gage https://en.wikipedia.org/wiki/Phineas_Gage
- rmorey 2y agoWe are so far ahead in the case of these models - we already have the complete wiring diagram! In biological systems we have only just begun to be able to create the complete neuronal wiring diagrams - currently worms, flies, perhaps soon mice
- bilsbie 2y agoHow are they handling attention in their approach? That’s going to completely change what features are looked at.
- tel 2y agoThey target the residual stream. Also they may have a definition of “feature” that’s more general than what you’re using. Consider reading their superposition work.
- bilsbie 2y agoIf anyone wants to team up and work on stuff like this (on toy models so we can run locally) please get in touch. (Email in profile) I’m so fascinated by this stuff but I’m having trouble staying motivated in this short attention span world.
- Mobil1 2y ago[dead]
- quotemstr 2y agoAt this risk of anthropomorphizing too much, I can't help but see parallels between the "my physical form is the Golden Gate Bridge" screenshot and the https://en.wikipedia.org/wiki/God_helmet https://en.wikipedia.org/wiki/God_helmet in humans --- both cognitive distortions caused by targeted exogenous neural activation.
- bjterry 2y agoIt would be interesting to allow users of models to customize inference by tweaking these features, sort of like a semantic equalizer for LLMs. My guess is that this wouldn't work as well as fine-tuning, since that would tweak all the features at once toward your use case, but the equalizer would require zero training data. The prompt itself can trigger the features, so if you say "Try to weave in mentions of San Francisco" the San Francisco feature will be more activated in the response. But having a global equalizer could reduce drift as the conversation continued, perhaps?
- ericflo 2y agoRelated: https://vgel.me/posts/representation-engineering/ https://vgel.me/posts/representation-engineering/
- bjterry 2y agoThanks!
- kromem 2y agoAt least for right now this approach would in most cases still be like using a shotgun instead of a scalpel. Over the next year or so I'm sure it will refine enough to be able to be more like a vector multiplier on activation, but simply flipping it on in general is going to create a very 'obsessed' model as stated.
- pagekicker 2y agoThe article doesn't explain how users can exploit these features in UI or prompt. Does anyone have any insight on how to do so?
- CephalopodMD 2y agoThey explicitly aren't releasing any tools to do this with their models for safety reasons. But you could probably do it from scratch with one of the open models by following their methodology.
- parentheses 2y agoIt's interesting that they used this to manipulate models. I wonder if "intentions" can be found and tuned. That would have massive potential for use and misuse. I could imagine a villain taking a model and amplifying "the evil" using a similar technique.
- parentheses 2y agoI wonder how interpretability and training can interplay. Some examples: Imagine taking Claude, tweaking weights relevant to X and then fine tuning it on knowledge related to X. It could result in more neurons being recruited to learn about X. Imagine performing this during training to amplify or reduce the importance of certain topics. Train it on a vast corpus, but tune at various checkpoints to ensure the neural network's knowledge distribution skews. This could be a way to get more performance from MoE models. I am not an expert. Just putting on my generalist hat here. Tell me I'm wrong because I'd be fascinated to hear the reasons.
- sanxiyn 2y agoSomeone should do this for Llama 3.
- e63f67dd-065b 2y agoI find Anthorpic's work on mech interp fascinating in general. Their initial towards monosemanticity paper was highly surprising, and so is this with the ability to scale to a real production-scale LLM. My observation is, and this may be more philosophical than technical: this process of "decomposing" middle-layer activations with a sparse autoencoder -- is it capturing accurately underlying features in the latent space of the network, or are we drawing order from chaos, imposing monosemanticity where there aren't any? Or to put it another way, were the features always there, learnt by training, or are we doing post-hoc rationalisations -- where the features exist because that's how we defined the autoencoders' dictionaries, and we learn only what we wanted to learn? Are the alien minds of LLMs truly also operating on a similar semantic space as ours, or are we reading tea leaves and seeing what we want to see? Maybe this distinction doesn't even make sense to begin with; concepts are made by man, if clamping one of these features modifies outputs in a way that is understandable to humans, it doesn't matter if it's capturing some kind of underlying cluster in the latent space of the model. But I do think it's an interesting idea to ponder.
- refulgentis 2y agoI'm allergic to latent space because I've yet to find any meaning to it beyond poetics, I develop an acute allergy when it's explicitly related to visually dimensional ideas like clustering. I'll make a probably bad analogy: does your mindmap place things near each other like my mindmap? To which I'd say, probably not, mindmaps are very personal, and the more complex we put on ours, the more personal and arbitrary they would be, and the less import the visuals would have ex. if we have 3 million things on both our mindmaps, it's peering too closely to wonder why you put mcdonalds closer to kids food than restaurants, and you have restaurants in the top left, whereas I put it closer to kids foods, in the top mid left.
- anentropic 2y agowhat if you averaged over millions of peoples' mindmaps?
- TeMPOraL 2y agoWhy would that matter? The absolute orientation of the mind map doesn't matter - maybe my map is actually very close to yours, subject to some rotation and mirroring? More than that, I'd think a better 2D analogy for the latent space is a force-directed graph that you keep shaking as you add things to it. It doesn't seem unlikely for two such graphs, constructed in different order, to still end up identical in the end. Thirdly: > if we have 3 million things on both our mindmaps, it's peering too closely to wonder why you put mcdonalds closer to kids food than restaurants, and you have restaurants in the top left, whereas I put it closer to kids foods, in the top mid left. In 2D analogy, maybe, but that's because of limited space. In 20 000 D analogy, there's no reason for our mind maps to meaningfully differ here; there's enough dimensions that terms can be close to other terms for any relationship you could think of.
- gautomdas 2y agoI've really been enjoying their series on mech interp, does anyone have any other good recs?
- PoignardAzur 2y ago"Transformers Represent Belief State Geometry in their Residual Stream": https://www.lesswrong.com/posts/gTZ2SxesbHckJ3CkF/transformers-represent-belief-state-geometry-in-their https://www.lesswrong.com/posts/gTZ2SxesbHckJ3CkF/transforme... Basically finding that transformers don't just store a world-model as in "what does the world that produce the observed inputs look like?", they store a "Mixed-State Presentation", basically a weighted set of possible worlds that produce the observed inputs.
- kromem 2y agoThe Othello-GPT and Chess-GPT lines of work. Was the first research work that clued me into what Anthropic's work today ended up demonstrating.
- kromem 2y agoGreat work as usual. I was pretty upset seeing the superalignment team dissolve at OpenAI, but as is typical for the AI space, the news of one day was quickly eclipsed by the next day. Anthropic are really killing it right now, and it's very refreshing seeing their commitment to publishing novel findings. I hope this finally serves as the nail in the coffin on the "it's just fancy autocomplete" and "it doesn't understand what it's saying, bro" rhetoric.
- Workaccount2 2y ago> on the "it's just fancy autocomplete" and "it doesn't understand what it's saying, bro" rhetoric. No matter what, there will always be a group of people saying that. The power and drive of the brain to convince itself that it is weaved of magical energy on a divine substrate shouldn't be underestimated. Especially when media plays so hard into that idea (the robots that lose the war because they cannot overcome love, etc.) because brains really love being told they are right. I am almost certain that the first conscious silicon (or whatever material) will be subjected to immense suffering until a new generation that can accept the human brains banality can move things forward.
- ben_w 2y agoIt tickles me somewhat to note that people using the phrase "stochastic parrot" are demonstrating in themselves the exact behaviour for which they are dismissive of the LLMs. > I am almost certain that the first conscious silicon (or whatever material) will be subjected to immense suffering until a new generation that can accept the human brains banality can move things forward. Indeed, though as we don't know what we're doing (and have 40 definitions of "consciousness" and no way to test for qualia), I would add that the first AI we make with these properties, will likely suffer from every permutation of severe and mild mental heath disorder that is logically possible, including many we have no word for because they would be incompatible with life if found in an organic brain.
- astrange 2y agoI think the research is good, but it's disappointing that they hype it by claiming it's going to help their basically entirely fictional "AI safety" project, as if the bits in their model are going to come alive and eat them.
- gdiamos 2y agoIt looks like Anthropic is now leading the charge on safety
- maherbeg 2y agoThey always were given that is a part of their mission.
- maciejgryka 2y agoI recorded myself trying to read through and understand the high-level of this if anyone's interested in following along: https://maciej.gryka.net/papers-in-public/#scaling-monosemanticity https://maciej.gryka.net/papers-in-public/#scaling-monoseman...