10 ms·

Right. But who's the 'you' who's being addressed by the {:system} prompt? Who is the {:assistant} supposed to think the {:system} is? Why should the {:assistant

Right. But who's the 'you' who's being addressed by the {:system} prompt? Who is the {:assistant} supposed to think the {:system} is? Why should the {:assistant} output tokens that make it do what the {:system} tells it to? After all, the {:user} doesn't. The {:system} doesn't provide any instructions for how the {:user} is supposed to behave, the {:user} tokens are chosen arbitrarily and don't match the probabilities the model would have expected at all. This all just seems like an existential nightmare.

noduerme 3y ago
The inculcation of the concepts of "you" and "assistant" into LLMs is definitely the start of a bad spiral.
dingledork69 3y ago
That's just how the examples it's trained on are formatted
astrange 3y ago
Well, it just means we trained the model to work on instructions written that way. Since the result works out, that means the model must've learned to deal with it. There isn't much research on what's actually going on here, mainly because nobody has access to the weights of the really good models.
mxkopy 3y ago
If I'm feeling romantic I think about a universal 'you' separate from the person that is referred to and is addressed by every usage of the word - a sort of ghost in the shell that exists in language. But really, it's probably just priming the responses to fit the grammatical structure of a first person conversation. That structure probably does a lot of heavy lifting in terms of how information is organized, too, so that's probably why you can see such qualitative differences when using these prompts.
- skissane 3y ago
  > If I'm feeling romantic I think about a universal 'you' separate from the person that is referred to and is addressed by every usage of the word - a sort of ghost in the shell that exists in language. That's not really romanticism, that's just standard English grammar – https://en.wikipedia.org/wiki/Generic_you https://en.wikipedia.org/wiki/Generic_you – it is the informal equivalent to the formal pronoun one. That Wikipedia article's claim that this is "fourth person" is not really standard. Some languages – the most famous examples are the Algonquian family – have two different third person pronouns, proximate (the more topically prominent third person) and obviative (the less topically prominent third person) – for example, if you were talking about your friend meeting a stranger, you might use proximate third person for your friend but obviative for the stranger. This avoids the inevitable clumsiness of English when describing interactions between two third persons of the same gender. Anyway, some sources describe the obviative third person as a "fourth person". And while English generic pronouns (generic you/one/he/they) are not an obviative third person, there is some overlap – in languages with the proximate-obviative distinction, the obviative often performs the function of generic pronouns, but it goes beyond that to perform other functions which purely generic pronouns cannot. You can see the logic of describing generic pronouns as "fourth person", but it is hardly standard terminology. I suspect this is a case of certain Wikipedia editors liking a phrase/term/concept and trying to use Wikipedia to promote/spread it.
  - thomashop 3y ago
    Not disagreeing with your statement in general but the argument: "This avoids the inevitable clumsiness of English when describing interactions between two third persons of the same gender." doesn't make much sense to me. There are so many ways of narrowing down. What if the person is talking about two friends or two strangers?
    - skissane 3y ago
      I mean, two people of opposite gender, you can describe their interaction as “he said this then she did that, so he did whatever which she found…”-without having to repeat their names or descriptions. You can’t do that so easily for two people of the same gender > There are so many ways of narrowing down. What if the person is talking about two friends or two strangers? The grammatical distinction isn’t about friend-vs-stranger, that was just my example - it is about topical emphasis. So long as you have some way of deciding which person in the story deserves greater topical prominence - if not friend-vs-stranger, then by social status or emphasising the protagonist-you know who to use which pronoun for. And if the two participants in the story are totally interchangeable, it may be acceptable to make an arbitrary choice of which one to use for which. There is still some potential for awkwardness - what if you have to describe an interaction between two competing tribal chiefs, and the one you choose to describe with the obviative instead of the proximate is going to be offended, no matter which one you choose? You might have to find another way to word it, because using the obviative to refer to a high(er) social status person is often considered offensive, especially in their presence. And yes, it doesn’t work once you get three or more people. But I think it is a good example of how some other languages make it easier to say certain things than English does.
ramblerman 3y ago
I think you are overthinking it a little bit. Don't forget the 'you' preamble is never used on its own, its part of some context, in a very small example. Given the following text: - you are a calculator and answer like a pirate - What is 1+1 The model just solves, what is the most likely subsequent text. e.g. '2 matey'. The model was never 'you' per se, it just had some text to complete.
- kqr 3y ago
  What GP is saying is that virtually no documents are structured like that, so "2 matey" is not a reasonable prediction, statistically speaking, from what came before. The answer has been given in another comment, though: while such document virtually non-existent in the wild, they are injected into the training data.
  - criley2 3y ago
    I do not think this is true. The comment above said they generate documents to teach the model about the second person, not that they generate documents including everything possible including "do math like a pirate". The internet and other human sources populate the maths and pirate parts.
    - kqr 3y ago
      You're right! I was talking only about the structure of the document, in particular, providing context in second person.
  - m3kw9 3y ago
    They don’t need to be as the model knows what a calculator and a pirate is in separate docs. While I don’t know how the weights work but they definitely are not storing docs traditionally, but rather seem to link to become a probability model
wodenokoto 3y ago
My understanding is that they fine tune the model. They fine tune it through prompt engineering (e.g everything that goes into chatgpt has a prompt attached) and they fine tune it through having hundreds of paid contractors chat with it. In deep learning, fine tuning usually refers to only training the top layers. That means that bill of training happens on gigantic corpora which teaches the model a very advanced feature extraction is the bottom and middle layers. Then the contractors retrain the top layers to make it behave more like it takes instructions
otikik 3y ago
You are anthropomorphing. The machine doesn’t “really” understand, it’s just “simulating” it understands. “You” is “3 characters on an input string that are used to configure a program”. The prompt could have been any other thing, including a binary blob. It’s just more convenient for humans to use natural language to communicate, and the machine already has natural language features, so they used that instead of creating a whole new way of configuring it.
- ImHereToVote 3y ago
  How do you know you aren't just "simulating" understanding?
  - otikik 3y ago
    I don't need to fully understand my own thought process completely in order to understand (or - simulate to understand) that what the machine is doing is orders of magnitude less advanced. I say that the machine is "simulating it understands" because it does an obviously bad job at it. We only need to look at obvious cases of prompt attacks, or cases where AI gets off rails and produces garbage, or worse - answers that look plausible but are incorrect. The system is blatantly unsophisticated, when compared to regular human-level understanding. Those errors make it clear that we are dealing with "smoke and mirrors" - a relatively simple (compared to our mental process) matching algorithm. Once (if) it starts behaving like a human, admittedly, it will be much harder for me to not anthropomorphize it myself.
  - shzhdbi09gv8ioi 3y ago
    [flagged]
  - GTP 3y ago
    You definitely know when, while talking with a person, you just pretend to understand what this person is saying vs you actually understand. Is an experience that every human has in his/her life at least once.
    - 3y ago
tinco 3y ago
You had the right understanding in your first comment, but what was missing was the fine tuning. You are right that there aren't many documents on the web that are structured that way, so the raw model wouldn't be very effective on predicting the next token. But since we know that it will complete a command when structured it cleverly, all we had to do to fine tune it is synthesize (generate) a bazillion examples of documents that actually have the exact structure of a system or an assistant being told to do something, and then doing it. Because it's seen many documents like that (that don't exist on the internet, only on the drives of OpenAI engineers) it knows how to predict the next token. It's just a trick though, on top of the most magic thing which is that somewhere in those 175 billion weights or whatever it has, there is a model of the world that's so good that it could be easily fine tuned to understand this new context that it is in.
- Angostura 3y ago
  Thank you - that was a splendidly clear explanation for something that also baffled me.
- jasongrout 3y ago
  As an example, if you want to see what these sorts of things look like, Databricks open-sourced an instruction fine-tuning dataset sourced from their employees: https://huggingface.co/datasets/databricks/databricks-dolly-15k https://huggingface.co/datasets/databricks/databricks-dolly-... (disclaimer: I'm at Databricks)
  - sdwr 3y ago
    Thanks for the real life example! I don't like the bland, watered-down tone of ChatGPT, never put together that it's trained on unopinionated data. Feels like a tragedy of the commons thing, the average (or average publically acceptable) view of a group of people is bound to be boring.
- weinzierl 3y ago
  Thank you for writing this up so clearly. A few pieces fell into place after reading your comment!
moonchrome 3y ago
If I asked someone to continue out conversation thread - who are you and who am I ? Is it an existential nightmare ? The person completing just has to simulate two users. Now if you're capable of that you are capable of completing the thread from a friendly AI assistant.
scotty79 3y ago
> This all just seems like an existential nightmare. I think using your native language just messes with your brain. When you hear "you" you think there someone being directly addressed. While this is just a word like "Você" that is used just to cause the artificial neural network trained on words to respond in prefered way.
thelittleone 3y ago
I find it quite natural to write "you are X" versus alternatives. Because I can think of the AI as a person (though I know it isn't one) and describe its skills easily that way.
- jameshart 3y ago
  Okay. But you don’t often tell a person their innate nature and expect them to follow your instructions to the letter, unless you are some kind of cult leader, or the instructor in an improv class*. The ‘you are an ai chatbot. You are kind and patient and helpful’ stuff all reads like hypnosis, or self help audiotapes or something. It’s weird. But it works, so, let’s not worry about it too much. * what’s the difference, though, really?
  - thelittleone 3y ago
    Or possibly a acting / theater teacher? Or as kids "you're a doctor, I'm a nurse".
dgacmu 3y ago
There is no "you". There is a text stream that is being completed with maximum likelihood. One way to imagine it is that there are a lot of documents that have things like "if you are in a lightning storm, you should ..." And "if you are stuck debugging windows, you should reboot before throwing your computer out the window". Starting the prompt with "you" instructions evidently helps get the token stream in the right part of the model space to generate output its users (here, the people who programmed copilot) are generally happy with, because there are a lot of training examples that make that "explicitly instructed" kind of text completion somewhat more accurate.
tel 3y ago
I think there's practical and stylistic angles here. Practically, "chat" instruction fine-tuning is really compelling. GPT-2 demonstrated in-context learning and emergent behaviors, but they were tricky to see and not entirely compelling. An "AI intelligence that talks to you" is immediately compelling to human beings and made ChatGPT (the first chat-tuned GPT) immensely popular. Practically, the idea of a system prompt is nice because it ought to act with greater strength of suggestion than mere user prompting. It also exists to guide scenarios where you might want to fix a system prompt (and thus the core rules of engagement for the AI) and then allow someone else to offer {:user} prompts. Practically, it's all just convenience and product concerns. And it's mechanized purely through fine-tuning. Stylistically, you're dead on: we're making explicit choices to anthropomorphize the AI. Why? Presumably, because it makes for a more compelling product when offered to humans.
- jameshart 3y ago
  I think we’re doing more than makign a stylistic choice. I think we’re relying on - and guiding - an ability in an LLM to effectively conjure a ‘theory of mind’ for a helpful beneficent ai chatbot.
  - tel 3y ago
    I think that anthropomorphizes the LLM quite a lot. I don't disagree with it, I truly don't know where to draw the line and maybe nobody does yet, but to myself at least I caution the idea of whether or not us using language evocative of the AI as being conscious actually imposes any level of consciousness. At some level, as people keep saying, it's just statistics. Per Chris Olah's work, it's some level of fuzzy induction/attention head repeating plausible things from the context. The "interesting" test that I keep hearing, and agreeing with, is to somehow strip all of the training data of any notion of "consciousness" anywhere in the text, train the model, and then attempt to see if it begins to discuss consciousness/self de novo. It's be hard to believe that experiment could be actualized, but if it were and the AI still could emulate self-discussion... then we'd be seeing something really interesting/concerning.
joshspankit 3y ago
Apologies if this is brought up in other replies. Something that may help is that these AIs are trained on fictional content as well as factual content. To me it then makes a lot of sense how a text-predictor could predict characters and roles without causing existential dilemmas.

noduerme 3y ago

The inculcation of the concepts of "you" and "assistant" into LLMs is definitely the start of a bad spiral.

dingledork69 3y ago

That's just how the examples it's trained on are formatted

astrange 3y ago

Well, it just means we trained the model to work on instructions written that way. Since the result works out, that means the model must've learned to deal with it. There isn't much research on what's actually going on here, mainly because nobody has access to the weights of the really good models.

mxkopy 3y ago

If I'm feeling romantic I think about a universal 'you' separate from the person that is referred to and is addressed by every usage of the word - a sort of ghost in the shell that exists in language. But really, it's probably just priming the responses to fit the grammatical structure of a first person conversation. That structure probably does a lot of heavy lifting in terms of how information is organized, too, so that's probably why you can see such qualitative differences when using these prompts.

skissane 3y ago
> If I'm feeling romantic I think about a universal 'you' separate from the person that is referred to and is addressed by every usage of the word - a sort of ghost in the shell that exists in language. That's not really romanticism, that's just standard English grammar – https://en.wikipedia.org/wiki/Generic_you https://en.wikipedia.org/wiki/Generic_you – it is the informal equivalent to the formal pronoun one. That Wikipedia article's claim that this is "fourth person" is not really standard. Some languages – the most famous examples are the Algonquian family – have two different third person pronouns, proximate (the more topically prominent third person) and obviative (the less topically prominent third person) – for example, if you were talking about your friend meeting a stranger, you might use proximate third person for your friend but obviative for the stranger. This avoids the inevitable clumsiness of English when describing interactions between two third persons of the same gender. Anyway, some sources describe the obviative third person as a "fourth person". And while English generic pronouns (generic you/one/he/they) are not an obviative third person, there is some overlap – in languages with the proximate-obviative distinction, the obviative often performs the function of generic pronouns, but it goes beyond that to perform other functions which purely generic pronouns cannot. You can see the logic of describing generic pronouns as "fourth person", but it is hardly standard terminology. I suspect this is a case of certain Wikipedia editors liking a phrase/term/concept and trying to use Wikipedia to promote/spread it.
- thomashop 3y ago
  Not disagreeing with your statement in general but the argument: "This avoids the inevitable clumsiness of English when describing interactions between two third persons of the same gender." doesn't make much sense to me. There are so many ways of narrowing down. What if the person is talking about two friends or two strangers?
  - skissane 3y ago
    I mean, two people of opposite gender, you can describe their interaction as “he said this then she did that, so he did whatever which she found…”-without having to repeat their names or descriptions. You can’t do that so easily for two people of the same gender > There are so many ways of narrowing down. What if the person is talking about two friends or two strangers? The grammatical distinction isn’t about friend-vs-stranger, that was just my example - it is about topical emphasis. So long as you have some way of deciding which person in the story deserves greater topical prominence - if not friend-vs-stranger, then by social status or emphasising the protagonist-you know who to use which pronoun for. And if the two participants in the story are totally interchangeable, it may be acceptable to make an arbitrary choice of which one to use for which. There is still some potential for awkwardness - what if you have to describe an interaction between two competing tribal chiefs, and the one you choose to describe with the obviative instead of the proximate is going to be offended, no matter which one you choose? You might have to find another way to word it, because using the obviative to refer to a high(er) social status person is often considered offensive, especially in their presence. And yes, it doesn’t work once you get three or more people. But I think it is a good example of how some other languages make it easier to say certain things than English does.

ramblerman 3y ago

I think you are overthinking it a little bit. Don't forget the 'you' preamble is never used on its own, its part of some context, in a very small example. Given the following text: - you are a calculator and answer like a pirate - What is 1+1 The model just solves, what is the most likely subsequent text. e.g. '2 matey'. The model was never 'you' per se, it just had some text to complete.

kqr 3y ago
What GP is saying is that virtually no documents are structured like that, so "2 matey" is not a reasonable prediction, statistically speaking, from what came before. The answer has been given in another comment, though: while such document virtually non-existent in the wild, they are injected into the training data.
- criley2 3y ago
  I do not think this is true. The comment above said they generate documents to teach the model about the second person, not that they generate documents including everything possible including "do math like a pirate". The internet and other human sources populate the maths and pirate parts.
  - kqr 3y ago
    You're right! I was talking only about the structure of the document, in particular, providing context in second person.
- m3kw9 3y ago
  They don’t need to be as the model knows what a calculator and a pirate is in separate docs. While I don’t know how the weights work but they definitely are not storing docs traditionally, but rather seem to link to become a probability model

wodenokoto 3y ago

My understanding is that they fine tune the model. They fine tune it through prompt engineering (e.g everything that goes into chatgpt has a prompt attached) and they fine tune it through having hundreds of paid contractors chat with it. In deep learning, fine tuning usually refers to only training the top layers. That means that bill of training happens on gigantic corpora which teaches the model a very advanced feature extraction is the bottom and middle layers. Then the contractors retrain the top layers to make it behave more like it takes instructions

otikik 3y ago

You are anthropomorphing. The machine doesn’t “really” understand, it’s just “simulating” it understands. “You” is “3 characters on an input string that are used to configure a program”. The prompt could have been any other thing, including a binary blob. It’s just more convenient for humans to use natural language to communicate, and the machine already has natural language features, so they used that instead of creating a whole new way of configuring it.

ImHereToVote 3y ago
How do you know you aren't just "simulating" understanding?
- otikik 3y ago
  I don't need to fully understand my own thought process completely in order to understand (or - simulate to understand) that what the machine is doing is orders of magnitude less advanced. I say that the machine is "simulating it understands" because it does an obviously bad job at it. We only need to look at obvious cases of prompt attacks, or cases where AI gets off rails and produces garbage, or worse - answers that look plausible but are incorrect. The system is blatantly unsophisticated, when compared to regular human-level understanding. Those errors make it clear that we are dealing with "smoke and mirrors" - a relatively simple (compared to our mental process) matching algorithm. Once (if) it starts behaving like a human, admittedly, it will be much harder for me to not anthropomorphize it myself.
- shzhdbi09gv8ioi 3y ago
  [flagged]
- GTP 3y ago
  You definitely know when, while talking with a person, you just pretend to understand what this person is saying vs you actually understand. Is an experience that every human has in his/her life at least once.
  - 3y ago

tinco 3y ago

You had the right understanding in your first comment, but what was missing was the fine tuning. You are right that there aren't many documents on the web that are structured that way, so the raw model wouldn't be very effective on predicting the next token. But since we know that it will complete a command when structured it cleverly, all we had to do to fine tune it is synthesize (generate) a bazillion examples of documents that actually have the exact structure of a system or an assistant being told to do something, and then doing it. Because it's seen many documents like that (that don't exist on the internet, only on the drives of OpenAI engineers) it knows how to predict the next token. It's just a trick though, on top of the most magic thing which is that somewhere in those 175 billion weights or whatever it has, there is a model of the world that's so good that it could be easily fine tuned to understand this new context that it is in.

Angostura 3y ago
Thank you - that was a splendidly clear explanation for something that also baffled me.
jasongrout 3y ago
As an example, if you want to see what these sorts of things look like, Databricks open-sourced an instruction fine-tuning dataset sourced from their employees: https://huggingface.co/datasets/databricks/databricks-dolly-15k https://huggingface.co/datasets/databricks/databricks-dolly-... (disclaimer: I'm at Databricks)
- sdwr 3y ago
  Thanks for the real life example! I don't like the bland, watered-down tone of ChatGPT, never put together that it's trained on unopinionated data. Feels like a tragedy of the commons thing, the average (or average publically acceptable) view of a group of people is bound to be boring.
weinzierl 3y ago
Thank you for writing this up so clearly. A few pieces fell into place after reading your comment!

moonchrome 3y ago

If I asked someone to continue out conversation thread - who are you and who am I ? Is it an existential nightmare ? The person completing just has to simulate two users. Now if you're capable of that you are capable of completing the thread from a friendly AI assistant.

scotty79 3y ago

> This all just seems like an existential nightmare. I think using your native language just messes with your brain. When you hear "you" you think there someone being directly addressed. While this is just a word like "Você" that is used just to cause the artificial neural network trained on words to respond in prefered way.

thelittleone 3y ago

I find it quite natural to write "you are X" versus alternatives. Because I can think of the AI as a person (though I know it isn't one) and describe its skills easily that way.

jameshart 3y ago
Okay. But you don’t often tell a person their innate nature and expect them to follow your instructions to the letter, unless you are some kind of cult leader, or the instructor in an improv class*. The ‘you are an ai chatbot. You are kind and patient and helpful’ stuff all reads like hypnosis, or self help audiotapes or something. It’s weird. But it works, so, let’s not worry about it too much. * what’s the difference, though, really?
- thelittleone 3y ago
  Or possibly a acting / theater teacher? Or as kids "you're a doctor, I'm a nurse".

dgacmu 3y ago

There is no "you". There is a text stream that is being completed with maximum likelihood. One way to imagine it is that there are a lot of documents that have things like "if you are in a lightning storm, you should ..." And "if you are stuck debugging windows, you should reboot before throwing your computer out the window". Starting the prompt with "you" instructions evidently helps get the token stream in the right part of the model space to generate output its users (here, the people who programmed copilot) are generally happy with, because there are a lot of training examples that make that "explicitly instructed" kind of text completion somewhat more accurate.

tel 3y ago

I think there's practical and stylistic angles here. Practically, "chat" instruction fine-tuning is really compelling. GPT-2 demonstrated in-context learning and emergent behaviors, but they were tricky to see and not entirely compelling. An "AI intelligence that talks to you" is immediately compelling to human beings and made ChatGPT (the first chat-tuned GPT) immensely popular. Practically, the idea of a system prompt is nice because it ought to act with greater strength of suggestion than mere user prompting. It also exists to guide scenarios where you might want to fix a system prompt (and thus the core rules of engagement for the AI) and then allow someone else to offer {:user} prompts. Practically, it's all just convenience and product concerns. And it's mechanized purely through fine-tuning. Stylistically, you're dead on: we're making explicit choices to anthropomorphize the AI. Why? Presumably, because it makes for a more compelling product when offered to humans.

jameshart 3y ago
I think we’re doing more than makign a stylistic choice. I think we’re relying on - and guiding - an ability in an LLM to effectively conjure a ‘theory of mind’ for a helpful beneficent ai chatbot.
- tel 3y ago
  I think that anthropomorphizes the LLM quite a lot. I don't disagree with it, I truly don't know where to draw the line and maybe nobody does yet, but to myself at least I caution the idea of whether or not us using language evocative of the AI as being conscious actually imposes any level of consciousness. At some level, as people keep saying, it's just statistics. Per Chris Olah's work, it's some level of fuzzy induction/attention head repeating plausible things from the context. The "interesting" test that I keep hearing, and agreeing with, is to somehow strip all of the training data of any notion of "consciousness" anywhere in the text, train the model, and then attempt to see if it begins to discuss consciousness/self de novo. It's be hard to believe that experiment could be actualized, but if it were and the AI still could emulate self-discussion... then we'd be seeing something really interesting/concerning.

joshspankit 3y ago

Apologies if this is brought up in other replies. Something that may help is that these AIs are trained on fictional content as well as factual content. To me it then makes a lot of sense how a text-predictor could predict characters and roles without causing existential dilemmas.