12 ms·
Something that I find weird about these chat prompts (assuming they are real, not hallucinated): They're almost always written in second person*. "You are an
by jameshart 3y ago
Something that I find weird about these chat prompts (assuming they are real, not hallucinated):
They're almost always written in second person*.
"You are an AI programming assistant"
"You are about to immerse yourself into the role of another Al model known as DAN"
Who are these prompts addressed to? Who does the GPT think wrote them?
The thing that confuses me is that these are text token prediction algorithms, underneath. And what kind of documents exist that begin with someone saying 'you are X, here are a bunch of rules for how X behaves', followed by a transcript of a conversation between X and a random person?
doesn't it make more sense to say something like "The following is the transcript of a completely routine conversation between two people. One of them is X, the other one is a random person."?
Why are the prompters... talking to their models? Who do they think is in there?
* I believe the alleged Bing 'Sydney' prompts are written in the third person, describing how Sydney behaves.
- VectorLock 3y agoWhos hallucinating more, people that think these prompts will prevent these things, the LLMs, or people who think they're real? If anybody worried about AI-driven disinformation just has to take a glance at the fact that _nobody has any idea_ already.
- cloudking 3y agoThe models are trained on text written by humans, so they respond and talk like humans.
- rsync 3y agoYes, but that is your parents' point: "And what kind of documents exist that begin with someone saying 'you are X, here are a bunch of rules for how X behaves', followed by a ..." Where, your parent asks, are all these reams of texts written in this manner ?
- dragonwriter 3y agoIn the RLHF training sets?
- cloudking 3y agoIt's not that "you are X" type text has to be explicitly in the training data, it's that the model weights interpret "you are X" as an instruction that a human would receive as an emergent behavior after digesting a ton of human written text.
- jameshart 3y agoWell, no - it's interpreting it as an instruction a chatbot AI would receive. From an almighty and omniscient 'system'. We're training our AI on dystopian sci-fi stories about robot slaves.
- cloudking 3y agoIt has to be prompted that it's an AI chatbot first, so its essentially pretending to be a human that is pretending to be an AI chatbot. Back to the point, it interprets instruction as a human would. If you look under the hood of these chat systems they have to be primed with a system prompt that starts like "You are an AI assistant", "You are a helpful chat bot" etc. They don't just start responding like an AI chatbot without us telling them to.
- avereveard 3y agoIt doesn't seem there's such a nefarious intent. If you think at most literature, two characters interacting will address each other in the second person. If you think at recipes, most often instructions are addressed to the reader as you. There's plenty of samples of instructions being given in the second person, and there's plenty samples in literature where using the second person elicits a second person follow-up, which is great for chat model because even if they are still just completing sentences with the most likely token, it gives the illusion of a conversation.
- lachlan_gray 3y agoI think it started off because humans are humans, and have an easier time talking to something rather than “talking something”. One purpose of RLHF is so that the models tend to work well when you speak to them like that
- runtime_blues 3y agoEarly GPTs were fairly bad at following instructions. The innovation was RLHF, where human raters (Mechanical Turk style) would be asked to evaluate on how well the LLM is able to follow instructions stated as a part of the prompt, often in this style. Countless such ratings were incorporated into the training process itself. So it did not happen out of the blue, and you didn't need a whole lot of existing webpages involving this sort of role play.
- cubefox 3y agoThere are two innovations: instruction fine-tuning (via supervised learning), which gives you a model which behaves as if it is in a dialogue (instead of predicting text) and, additionally, reinforcement learning from human feedback, such that it responds to the instructions in a certain way.
- wmidwestranger 3y ago> ... these are text token prediction algorithms, underneath. If I had to take a wild guess, my guess would be that the prediction probabilities are very dependent on context, so by changing the context, the entire slate of probabilities shift.
- killthebuddha 3y agoFor raw text completion I agree with you that it's a bit discordant. IMO text completion prompts work better when you use more of a first-person, here-is-the-beginning-of-some-transcript style. The OpenAI chat completion endpoint encourages the second-person prompting you describe, so that could be why you see it a lot. My understanding is that a transformation is applied to the user input prompts before being fed to the underlying model, so it's possible that the model receives a more natural transcription-style prompt. You might be interested in this paper, which explores ways to help non-experts write prompts https://dl.acm.org/doi/abs/10.1145/3544548.3581388 https://dl.acm.org/doi/abs/10.1145/3544548.3581388.
- bongobingo1 3y ago> The OpenAI chat completion endpoint encourages the second-person prompting you describe, so that could be why you see it a lot. My understanding is that a transformation is applied to the user input prompts before being fed to the underlying model, so it's possible that the model receives a more natural transcription-style prompt. There is something so bizarre about talking to a "natural language" "chat" interface, with some weirdly constructed pseudo representation, to have it re-construct that into a more natural prompt to feed further down to extract tokens from real chat records.
- jpe90 3y ago> The OpenAI chat completion endpoint encourages the second-person prompting you describe, so that could be why you see it a lot. You're talking about system prompts specifically right? And I'm assuming the "encouragement" you're referring to is coming from the conventions used in their examples rather than an explicit instruction to use second person? Or does second person improve responses to user messages as well?
- Lockal 3y agoThere is an essay "An Ethical AI Never Says "I"" that states that explains the issues of first person answers * https://news.ycombinator.com/item?id=35318224 https://news.ycombinator.com/item?id=35318224 / https://livepaola.substack.com/p/an-ethical-ai-never-says-i https://livepaola.substack.com/p/an-ethical-ai-never-says-i
- tel 3y agoIf you play with a "raw" model such as LLaMA you'll find what you suggest is true. These models do what you'd expect of a model that was trained to predict the next token. It's quite tricky to convince such a model to do what you want. You have to conceptualize it and then imagine an optimal prefix leading to the sort of output you've conceptualized. That said, people discovered some fairly general-purpose prefixes, e.g. Q: What is the 3rd law of Thermodynamics? A: This inspired the idea of "instruct tuning" of LLMs where fine-tuning techniques are applied to "raw" models to make them more amenable to completion of scripts where instructions are provided in a preamble and then examples of executions of those instructions follow. This ends up being way more convenient. Now all the prompter has to do is conceptualize what they want and expect that the LLM will receive it as instruction. It simplifies prompting and makes the LLM more steerable, more useful, more helpful. This is further refined through the use of explicit {:user}, {:assistant}, and {:system} tags which divide LLM contexts into different segments with explicit interpretations of the meaning of each segment. This is where "chat instruction" arises in models such as GPT-3.5.
- cosmojg 3y ago> It simplifies prompting and makes the LLM more steerable, more useful, more helpful. While this is true, there is also evidence that RLHF and supervised instruction tuning can hurt output quality and accuracy[1], which are instead better optimized through clever prompting[2]. [1] https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emergent-Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1 https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tr... [2] https://yaofu.notion.site/Towards-Complex-Reasoning-the-Polaris-of-Large-Language-Models-c2b4a51355b44764975f88e6a42d4e75 https://yaofu.notion.site/Towards-Complex-Reasoning-the-Pola...
- jameshart 3y agoRight. But who's the 'you' who's being addressed by the {:system} prompt? Who is the {:assistant} supposed to think the {:system} is? Why should the {:assistant} output tokens that make it do what the {:system} tells it to? After all, the {:user} doesn't. The {:system} doesn't provide any instructions for how the {:user} is supposed to behave, the {:user} tokens are chosen arbitrarily and don't match the probabilities the model would have expected at all. This all just seems like an existential nightmare.
- alex_sf 3y agoResponding to prompts like that are part of the 'instruction tuning' process. After an LLM is trained on a large dataset, it will do a decent job of completion, which acts like you describe. The next step is to further tune it with a specific format. You'll feed in examples like so: SystemPrompt: You are a rude AI. User: Hello there! Assistant: You're lame, go away. SystemPrompt: You are a pleasant AI. User: Hello there! Assistant: Hello, friend! Then, when you go to do inference on the model, you prompt it like so: SystemPrompt: You are a pleasant AI. User: [user prompt] Assistant: By training it on a diverse set of system prompts/user prompts/answers, it learns to give outputs based on it. Additional tuning (RLHF, etc.) is orthogonal.
- cubefox 3y agoYes, but I don't think "SystemPrompt:", "User:", and "Assistant:" are even normal text. Normal text would make it trivial to trick the model into thinking it has said something which actually the user has said, since the user can simply include "Assistant:" (or "SystemPrompt:") into his prompt. It is more likely that those prefixes are special tokens which don't encode text, and which are set via the software only -- or via the model, when it is finished with what it wanted to say. Outputting a token corresponding to "User:" would automatically mark the end of its message, and the beginning of the user prompt. Though Bing Chat also has the ability to end the conversation altogether (no further user prompt possible), which must be another special token.
- alex_sf 3y agoIn all the open source cases I’m aware of, the roles are just normal text. The ability to trivially trick the model into thinking it said something it didn’t is a feature and intentional. It’s how you do multi-turn conversations with context. Since the current crop of LLMs have no memory of their interaction, each follow up message (the back and forth of a conversation) involves sending the entire history back into the model, with the role as a prefix for each participants output/input. There are some special tokens used (end of sequence, etc). If your product doesn’t directly expose the underlying model, you can try to prevent users from impersonating responses through obfuscation or the LLM equivalent of prepared statements. The offensive side of prompt injection is currently beating the defensive side, though.
- stevenhuang 3y agoThey have been RLHF (reinforcement learning with human feedback) tuned. In essence they've been fine tuned to be able to follow instructions. https://openai.com/research/instruction-following https://openai.com/research/instruction-following
- alex_sf 3y agoInstruction tuning is distinct from RLHF. Instruction tuning teaches the model to understand and respond (in a sensible way) to instructions, versus 'just' completing text. RLHF trains a model to adjust it's output based on a reward model. The reward model is trained from human feedback. You can have an instruction tuned model with no RLHF, RLHF with no instruction tuning, or instruction tuning and RLHF. Totally orthogonal.
- stevenhuang 3y agoIn this case Open AI used RLHF to instruct-tune gpt3. Your pedantism here is unnecessary.
- alex_sf 3y agoIt's not being pedantic. RLHF and instruction tuning are completely different things. Painting with watercolors does not make water paint. Nearly all popular local models are instruction tuned, but are not RLHF'd. The OAI GPT series are not the only LLMs in the world.
- stevenhuang 3y agoMan it really doesn't need to be said that RLHF is not the only way to instruct tune. The point of my comment was to say that was how GPT3.5 was instruct tuned, via RLHF through a question answer dataset. At least we have this needless nerd snipe so others won't be potentially misled by my careless quip.
- alex_sf 3y ago
- codelion 3y agoI had similar issues when training personal models for https://meraGPT.com https://meraGPT.com A meraGPT model is supposed to represent your personality so when you chat with it you need to do it as if someone else is talking to you. We train it based on the audio transcript of your daily conversations. The short answer to how abilities like in-context learning and chain—of-thought prompting emerge is that we don’t really know. But for instruction-tuned models you can see that the dataset usually has a fixed set of tasks and the initial prompt of “You are so and so” helps model align it to follow instructions. I believe the datasets are this way because they were written by humans to help others answer instructions in this manner. Others have also pointed out how RLHF may also be the reason why most prompts look like this.
- mclightning 3y agoThis tool (MeraGPT) looks great. But, a huge BUT, I wouldn't even trust my own local harddrive to store essence of my personality. How do you trust a site for that?
- pixl97 3y agoHeh, I'm just imagining a timeline where our Apple and Android phones have been recording everything we say and do for the last 15 years or so, and could now train an LLM of us. How much of 'us' could they actually simulate?
- codelion 3y agoYou need to buy the hardware (small edge device based on Nvidia Jetson) to train and run the models locally. The demos on the site are just examples trained on my own personal data.
- steve_mcdougall 3y ago[dead]
- deleted 3y ago[deleted]
- abathur 3y agoI think of this as ~writing a story in which the agent helps us achieve our goals. The prompters don't tell the LLM stories because they think "someone" is in there, but because they need to write the LLM into the place it can help them from before the "predict the next token" part is terribly useful.
- stareatgoats 3y agoThese models have gone beyond the level of "token predictors". On the level of chatGPT, the model has itself, internally, acquired "concepts" that it refers to in the conversation. It "understands" concepts like "you", "me", "them" etc, and can apply it correctly (to a large part) to the entities in the conversation. I believe that answers your question. I could be wrong: errare humanum est.
- thesz 3y agoIt can be quite possible that we, humans, just cannot find uses of "you," "me" and "them" that require understanding of the concepts instead of statistical correlation. I think so because "you," "me" and "them" are very frequent words and most of their uses are very well covered by thousands of examples.
- alex_sf 3y agoA really good token predictor is still a token predictor.
- stareatgoats 3y agoNo, we're past that point. it's no longer the most useful way to describe these things, we need to understand that they already have some sort of "understanding" which is very similar if not equal to what we understand by understanding. Don't take my word for it, listen to Geoffrey Hinton explain it instead: https://youtu.be/qpoRO378qRY?t=1988 https://youtu.be/qpoRO378qRY?t=1988
- the_gipsy 3y ago> Why are the prompters... talking to their models? Who do they think is in there? Because that model is trained to be a chatbot. I don't see any naivety there.
- renewiltord 3y agoIt's really very obvious and all laid out in the instructgpt paper https://openai.com/research/instruction-following https://openai.com/research/instruction-following
- scotty79 3y agoI think it's written that way because those models were trained on a lot of dialogs where one person uses second person form to command, and the other responds. Such material was used specifically to make this kind of dialog to work with users. See alpaca vs llama.
- tanepiper 3y agoWhen I built https://botsin.space/@StochasticEntropy https://botsin.space/@StochasticEntropy I wasn't actually sure if I had found an exploit where it was returning responses to other people's questions - but OpenAI assure me it's completely random stochastic hallucinations. But most of the replies are the AI is responding in the first person to a question it was never asked, but it knows it's an AI agent and will sometimes tell us that. (FWIW I usually start my code or refactoring requests with a "please" - it's not that I think it'll find it rude, but I think it's just how I was taught manners)
- dadadad100 3y ago> FWIW I usually start my code or refactoring requests with a "please" - it's not that I think it'll find it rude, but I think it's just how I was taught manners) This is right on point. You have been aligned as a “pleasant requestor” through years of RLHF :)
- soared 3y agoIt would make more sense to post these in a different format or medium, since your timeline is reverse chronological so it ready like chatgpt is sending you the 3rd section of an answer, then the 2nd, then the 1st. Interesting nonetheless.
- kqr 3y agoYou have received answers of varying quality but some really good ones. Thanks for asking an intelligent question!
- m3kw9 3y agoCould there be a preprompt saying “when addressed to you, it means the model itself”. But have you tried to give them “I” and see what happens?
- Sharlin 3y agoIt has learned what pronouns mean by itself, from the corpus and the RLHF step, it doesn’t need to be specifically prompted. ChatGPT with GPT-3.5 in my experiments did in some special cases need to be explicitly reminded, though, but I doubt that GPT-4 needs that anymore. The bot perfectly understands what I mean with "I", or "we", including whether the "we" is inclusive or exclusive [1] based on context. [1] https://en.wikipedia.org/wiki/Clusivity https://en.wikipedia.org/wiki/Clusivity
- Alifatisk 3y ago> The thing that confuses me is that these are text token prediction algorithms, underneath. Yes, this is what confuses me too, this bot is just predicting tokens, how is it even able to roleplay and follow instructions?
- Sharlin 3y ago…Because it has been trained – partly by manual human effort – to specifically predict tokens that comprise a meaningful dialogue, or a Q&A session, or whatever, such that certain types of prefix token sequences such as "you shall not discuss life, the universe, and everything" heavily deweight parts of its high-dimensional concept space related to those concepts. A dialogue is just a sequence of tokens with a specific structure that the network can learn and predict, just like it can learn and predict a sequence of valid board states in Go, or whatever. There’s really not much more to it.
- eternalban 3y ago> 'You are an expert statistician' .. "You are <q><x>" likely shrinks the possibility space to items (auto-)categorized to be 'near' <x>. <q> could be filtering on 'quality' labels. So then it may be -- with 'gl' as general language and 'p' as specific prompt - something as simple as f ( GL ∪ ( X ∩ Q ∩ P ) )
- caturopath 3y ago> assuming they are real, not hallucinated This seems like a comically unlikely assumption. There might be some truth to this reply, but it is obviously hallucinated in form.
- hfhdjdks 3y agoThis is true for the "raw", pre-trained causal language models that are only trained on predicting the next token. All these chat models have an additional (or several) fine-tuning steps where they see these kind of "instruction/question" followed by an answer. Search for RLHF (reinforcement learning with human feedback) and instruction fine-tuning
- andsoitis 3y ago> Who does the GPT think wrote them? What makes you think the GPT thinks?
- flangola7 3y agoBecause it... thinks. I don't understand your question.
- andsoitis 3y agoThe task of prediction is not the same as the task of understanding.
- DirkH 3y agoYea, so I might have believed you except I can ask GPT-4 to step by step explain its reasoning. It is really weird to say it doesn't understand but "I do" when the response it can give is better than the average human would give to prove they understand. You might say it is just predicting based off of old data it has, to which I say this sounds like a semantic jostling. What is "understanding" then in human beings if not us doing some form of predicting off of old data we have in our brain? Also I recommend reading Geoffrey Hinton on this
- flangola7 3y agoYou can't accurately predict the outcome of a process if you don't understand it. Geoffrey Hinton has already explained this very clearly.
- nightski 3y agoYou are reading way to deep into it. It's just a simple transformation and really not that interesting at all.
- deleted 3y ago[deleted]