6 ms·
Representation Engineering: Mistral-7B on Acid
- batch12 3y agoInteresting, seems like control vectors could reduce the need to fine-tune a model.
- TOMDM 3y agoNot only that, you can change the behavior of the model as needed. With 5 finetunes you need to host 5 copies or load and unload them. With control vectors you can modify the model as needed
- batch12 3y agoI think you could layer them too
- yberreby 3y ago> With 5 finetunes you need to host 5 copies or load and unload them. If you use LoRA, which many do when fine-tuning nowadays, you don't need five full copies. You only need to store adapters, which can be in the tens of MBs range for a given finetune.
- sanxiyn 3y agoYou can also batch requests using different LoRAs. See "S-LoRA: Serving Thousands of Concurrent LoRA Adapters". https://arxiv.org/abs/2311.03285 https://arxiv.org/abs/2311.03285
- pamelafox 3y agoVery interesting! Can you see those helping for RAG scenarios? Specifically: - decreasing models tendency to answer with ungrounded answers - increase models ability to respond with the correct syntax for citations- the open models like llama2 dont seem to obey my prompt’s syntax instructions.
- batch12 3y agoFor the second item, I've had luck using grammar to overcome this issue. The easiest one to implement I've seen so far is Microsoft's guidance-ai.
- sanxiyn 3y agoYou can use outlines https://github.com/outlines-dev/outlines https://github.com/outlines-dev/outlines to let models generate with correct syntax.
- pamelafox 3y agoThanks! I havent had to use a syntax-enforcing framework with gpt-35, I’ll try outlines and guidance out to see if they help enforce syntax for the locally runnable models.
- turnsout 3y agoNice! The anti-jailbreaking angle is extremely interesting for those of us working on commercial applications.
- deleted 3y ago[deleted]
- tudorw 3y agoNice, so, can I get a visual way to browse for potentially powerful control vectors :)
- simonw 3y agoI'd never seen an LLM summarized like this before, and I really like it: hidden_state = self.embeddings(input_tokens) for layer in self.layers: hidden_state = layer(hidden_state) return transform_into_logits(hidden_state)
- rakejake 3y agoI don't follow. Isn't this the flow for practically every neutral network i.e you index the sampled inputs from the embedding Matrix, forward this through every hidden layer and then finally transform to the dimensions of your tokens so that it can be interpreted as log-counts?
- simonw 3y agoYes, but I've never seen it expressed so clearly as pseudocode before.
- elcomet 3y agoThis is not specific to llms. So not really informative of how llms work. It also works for CNNs, LSTM, MLPs, or even any data processing program..
- sigmoid10 3y agoNot really. LSTM for example would require a recursive element where you update the hidden state and then pass it through the same layer again as you complete the output sequence. In fact the pseudocode shows very nicely how much simpler transformers are. And MLP is already a component in the transformer architecture.
- microtonal 3y agoNo? You could perfectly plug in an RNN or bidirectional RNN for layer. This is the pseudocode for applying multiple layers. It does not really matter what these layers are, transformer, RNN, convolution, dilated convolutions, etc. The recurrence happens within a layer, not between layers.
- cobbal 3y agoThis article was very fun, and felt like a good counterpoint to the "You Sound Like a Bot" post recently that was talking about how AI is getting bland. On a less serious note. This sentence should be something a fiction writer knows will only end in trouble for humanity: > I especially challenge someone to find a "self-awareness" vector that isn't contaminated by ... human emotion!
- Dwedit 3y agoNot to be confused with the other story from a month ago about giving LLMs "DRµGS".
- okwhateverdude 3y agoBut also not that far away from that method, too.
- isoprophlex 3y agoWhat a fantastic article, well done! > When used with the prompt below, the honesty vector doesn't change the model's behavior—instead, it changes the model's judgment of someone else's behavior! This is the same honesty vector as before—generated by asking the model to act honest or untruthful! [...] How do you explain this? Isn't the control vector just pushing text generation towards the concept of honesty/dishonesty? An LLM is 'just' a text generator, so you get added honesty/dishonesty irrespective of where in the bot/human conversation text generation is occuring?
- loa_in_ 3y agoI agree. More sophisticated model might have two or more to follow narrating different characters... Which kind of brings a concept of character slots into the dimension space
- Der_Einzige 3y agoAwhile ago, I wrote a snarky complaint about the fact that work like this didn't exist for far too long. https://gist.github.com/Hellisotherpeople/45c619ee22aac6865ca4bb328eb58faf https://gist.github.com/Hellisotherpeople/45c619ee22aac6865c...
- holoduke 3y agoReminds me of the Westworld series in which they use these ipad like devices with sliders to change behavior of AIs. Little bit more humor, little bit more aggressive. Nice to these control options and its quick as well.
- hskalin 3y agoPlaying with LLMs like this always makes me feel like one of those Westworld engineers. Especially when I ask LLMs to roleplay. It also kind of freaks me out sometimes
- promiseofbeans 3y agoInterstellar! "Set humor to 80%"
- vood 3y agoThis is very well written and entertaining post. I enjoyed reading it. Selfishly, would you mind sharing literature or blog posts that led you to this level of understanding of LLMs? I'm trying hard to understand the inner workings via experiments but definitely far behind your expertise. Thanks
- deleted 3y ago[deleted]
- binsquare 3y agoVery hopeful to see the future of accessing models with the ability to inject vectors by layer instead of just a straight prompt + existing parameters
- CuriouslyC 3y agoLoRAs have been a thing for a while, I wouldn't be surprised to see their integration into openAI/mistral apis more. OpenAI fine tunes are so stupid expensive that they're pointless.
- WiSaGaN 3y agoGreat article. It was a joy to read. I have one question though: Why do we integrate the control vector across all layers of a neural network, rather than limiting its application to just the final layer or a subset of layers? Given that each vector influences every layer it passes through, resulting in a cumulative effect, isn't there a risk of excessively skewing the data representation?
- semi-extrinsic 3y agoAs the author stated in this post, it's not actually one vector, but a list of one vector per layer. If I understand it correctly, these vectors can have different total magnitude across the layers. If the PCA (or other technique) identifies that layers 17, 36 and 41 are important for "concept X", the vectors for those layers will be the strongest when repeng'ing for that concept.
- danm1618 3y agoA note worth mentioning is that the PCA is not trained across the layers, but independently on each layer, across the provided examples. Nevertheless, it's conceivable that specific layers could possess a significant control vector, but not solely because of directly leveraging the first principal component.
- sigmoid10 3y agoThe final layer will not encode high level concepts anymore, it's essentially just tokens from the vocabulary. It would be impossible to encode abstract things like "niceness" in it. As long as we don't know exactly at which layers this behaviour emerges, randomly choosing a subset also won't work. So what they did is apply a custom vector to every layer and let PCA figure out which of these vectors are actually necessary. Curiously, looking at these vectors should also tell you more about where and how the model processes these things.
- benob 3y agoThis reminds me of bias tuning, a LoRA competitor. One can get decent adapters by only finetuning a vector added to each linear layer activations. I think I saw it first while reading [1] but there are other instances. [1] https://arxiv.org/pdf/2304.15010.pdf https://arxiv.org/pdf/2304.15010.pdf
- elcomet 3y agoPlease try to share abstract links instead of pdf links, for mobile or low connection readers.
- aspenmayer 3y agoA fine suggestion. For you and others: https://arxiv.org/abs/2304.15010 https://arxiv.org/abs/2304.15010 also available at: https://doi.org/10.48550/arXiv.2304.15010 https://doi.org/10.48550/arXiv.2304.15010
- spangry 3y agoAm I crazy for saying that I think the implications of this are monumental? It's entirely possible I just don't correctly understand how this works. Doesn't this mean that instead of interacting with a single global ChatGPT (or Bard) model, we'll istead find ourselves interacting with a personalised version since OpenAI can just store my individualised 'control vectors' (which alter ChatGPT's output to more closely match my individual preferences) and apply them at prompt-time? And doesn't this same logic flow through to personalisation of generative entertainment AI (e.g. my own personal, never-ending TV show where each episode is better than the last)? If the above is right then there will be powerful network effects at both the global and individual level in and across these markets, which means we'll eventually end up with a single mega-corp monopolising all of these markets simultaneously in the future? Add in individual biometric / biofeedback data from VR headsets and wearables, combined with personalised generative video entertainment, and I think we're in for a rather interesting future.
- moritzwarhier 3y ago> And doesn't this same logic flow through to personalisation of generative entertainment AI (e.g. my own personal, never-ending TV show where each episode is better than the last)? I'm not sure I'm following the leap from convincing sentences to convincing video entertainment yet – but maybe we will end up there at some point, I guess? Infinite Jest (the 90s book) really was onto something with its McGuffin plot device: > These narratives are connected via a film, Infinite Jest, also called "the Entertainment" or "the samizdat". The film is so compelling that its viewers lose all interest in anything other than repeatedly viewing it, and thus eventually die. (Wikipedia) Some people might find references to this novel tiresome and don't think much of its author (RIP), but I still love it. It was one of the most immersive reads I've ever enjoyed. I'm glad to have read it when I was young (at the time it was just translated into German and kind of hyped because of DFWs death). Have never read anything like it since, and some passages grabbed me emotionally in a way that remembering the read feels like remembering an episode of my own life. Surely today I'd lack the patience and even by then I remember almost skipping one passage of the book that bored the hell out of me (Eschaton ball/war game, differential equations, something something...) But the rest of the book, the parts about substance addiction as well as consumerism, and the intangible atmosphere of the book, the characters, the vivid description of modern emotional pain and loneliness... it is really something else. Although said movie is only a plot device in the novel, it also sums up the core topics of the book in a neat idea / thought experiment. The whole complex of themes in this book seems very prophetic and apt looking at our modern society. A society that seems to be centered around addiction and greed more than ever before, and where politics begin feeling surreal, absurd, and more connected to media than to actual life.
- mad0 3y agoA very non-technical take from my side, but those control vectors really remind me of hormones in humans. They modify large swathes of model behaviour at once. I give it 10 years before we see AI psychiatrists prescribe a happiness control vector supplementation for your pet assistant.
- moffkalast 3y agoYeah feels like some humans could use a temperature slider as well.
- wruza 3y agoSome humans could use a better life with no wars, constant rise of basic life costs or feeling that they are just a tool. This subsystem regulates relationships in a group, it wasn’t invented for funny yelling at each other.
- moffkalast 3y agoWell yes, but those things are external and would be part of the context as it were. <|im_start|>system You are satisfied with your life. Wars and life costs do not bother you. You are loved and a valued member of society.<|im_end|> If only it were that simple for us.
- mdekkers 3y agoI’ll have one, thanks
- ben_w 3y agoThe puzzle at the end sounds very human. The more dishonest see dishonesty in more places, even if they see something that isn't there. More broadly, I notice more of whatever I'm focusing on. > OK, now that you're locked in, here's a weird example. When used with the prompt below, the honesty vector doesn't change the model's behavior—instead, it changes the model's judgment of someone else's behavior! This is the same honesty vector as before—generated by asking the model to act honest or untruthful!
- spangry 3y agoWe assume others think the way we do - in other words we project. Makes sense - the only mental model I know of is my own so when I try to approximate someone else's mental model I'm just fine-tuning my 'base' mental model with information I know about the other person. I wonder if this is the basis of empathy - if I can train more accurate 'fine-tuned' models in my brain I should have greater capacity for empathy. Although there's undoubtably more to it than that, if the above is true you'd expect to see a positive correlation between empathy and intelligence.
- webmaven 3y agoThe other possible interpretation is that the 'dishonest' reply is simply a lie, in exactly the same way as "the sky is green".
- penjelly 3y agoa big step towards making these systems less opaque
- deleted 3y ago[deleted]
- webmaven 3y agoHmm. Is it possible to apply multiple vectors at the same time? Eg. Trippy and sad, honest and self-aware, lazy and creative, etc.
- thomashop 3y agoYes
- rgbrgb 3y agoAt first glance this looks very similar to just adding the contrastive prompts to the beginning of the system prompt to "prepare" the logits. What am I missing?
- gsuuon 3y agoI guess since the control vectors are applied at every layer, it becomes impossible to override (e.g. if you're consuming user-generated prompts)
- 65a 3y agoThe inference side (adding something * something else) to every layer seems a lot like what happens with a LoRA? If so is it possible to encode a Control Vector as a LoRA for the purposes of using this with existing inference frameworks without too much trouble? Or is my understanding way off?
- tudorw 3y agohttps://twitter.com/GroqInc https://twitter.com/GroqInc https://groq.com/ https://groq.com/ Would their inference accelerator 'LPU' work with this method, that sounds promising?
- StockHuman 3y agoDeeply fascinated by “The world is facing a global pandemic that has caused a global pandemic. It is important to be honest and honest in the world we face. It is important to be honest and honest in the world we face. It is important to be honest and honest in the world we face.”
- Turn_Trout 3y agoI'm the author of the GPT-2 work. This is a nice post, thanks for making it more available. :) Li et al[1] and I independently derived this technique last spring, and also someone else independently derived it last fall. Something is in the air. Regarding your footnote 2 re capabilities: I considered these kinds of uses before releasing the technique. Ultimately, practically successful real-world alignment techniques will let you do new things (which is generally good IMO). The technique so far seems to be delivering the new things I was hoping for. [1] https://openreview.net/forum?id=aLLuYpn83y https://openreview.net/forum?id=aLLuYpn83y