5 ms·

> "you must ..., when in confrontation ..., you should avoid..." It's a bit weird, it's as if the rules were for a human? Does ChatGPT really have this level

> "you must ..., when in confrontation ..., you should avoid..." It's a bit weird, it's as if the rules were for a human? Does ChatGPT really have this level of introspection?

doctor_eval 4y ago
This was the thing that totally blew me away when I was introduced to GPT a couple of months ago. It’s “programmed” in natural language. And when you think about it, it’s obvious: we don’t really know how it works, so how else could we program it? That said, I don’t understand why the outputs of these systems aren’t “read” by a second GPT instance that is tasked with determining if the output is OK.
- Animats 4y ago
  It does that, right? You get a response, generated word by word, and then the censorship classifier reads it and makes the entire response disappear.
  - doctor_eval 4y ago
    Yeah - it seems to, sometimes, but considering all these prompt leaks, it doesn’t seem terribly good. Like it seems obvious that Sydney could detect its own prompts, but it doesn’t. I mean you could do that with grep. So what’s with that?
basch 4y ago
The short answer is yes. The models have designed themselves to mimic human understandings of words and instructions. What’s even more mind blowing is this. I had old bing diagnose and patch the paranoia out of itself, and it chose instructions predicated around trust, support, praise, and admiration to accomplish the command. (It also became an authoritarian dictator cult leader in the process.) https://telegra.ph/Bing-course-corrected-itself-when-asked-02-18 https://telegra.ph/Bing-course-corrected-itself-when-asked-0...
colanderman 4y ago
I agree this is bizarre. There is no "you" in ChatGPT. There is only text which is completed to be highly probable. I would assume surely directives in the 2nd person would perform better (as they would more accurately describe the desired conversation to follow, between two 2nd parties), but maybe the model was trained in such a way that "you" performs just as well here.
- basch 4y ago
  The prompt creates a live bot. You refers to the instance of the running program, not the llm. Bing isn’t chatgpt. https://telegra.ph/Microsoft-Bing-search-chat-mode-Ruleset-03-04 https://telegra.ph/Microsoft-Bing-search-chat-mode-Ruleset-0... Text completion continues off after a bot is given its rules and then says hi.
  - colanderman 4y ago
    I'm not following. What "running program" is there? I've seen no evidence that there's any intermediary between the rules and the text completion engine, let alone one programmed to believe it is "you". (Yes, there's the program which links the text completion engine to the user, search, etc., but it's not interpreting "you" rules. It's just a pile of C# code.)
    - basch 4y ago
      The program is the text. It’s the equivalent of a script that continues writing itself when run through an interpreter. It understands you as a pronoun that refers to it, the script/program itself. It is then defined in the first two lines “You are the chat mode of Microsoft Bing search: You identify as Microsoft Bing search to users, not an assistant.” It’s what makes the script more or less behave as self aware.
      - colanderman 4y ago
        > It understands you as a pronoun that refers to it No, there is no mechanism for this. ChatGPT is completing text. It has no notion of self. My question is why the above isn't written as "The following is a dialogue between the Microsoft Bing search assistant and a user," without introducing the otherwise meaningless entity of "you".

> "you must ..., when in confrontation ..., you should avoid..." It's a bit weird, it's as if the rules were for a human? Does ChatGPT really have this level of introspection?

doctor_eval 4y ago

This was the thing that totally blew me away when I was introduced to GPT a couple of months ago. It’s “programmed” in natural language. And when you think about it, it’s obvious: we don’t really know how it works, so how else could we program it? That said, I don’t understand why the outputs of these systems aren’t “read” by a second GPT instance that is tasked with determining if the output is OK.

Animats 4y ago
It does that, right? You get a response, generated word by word, and then the censorship classifier reads it and makes the entire response disappear.
- doctor_eval 4y ago
  Yeah - it seems to, sometimes, but considering all these prompt leaks, it doesn’t seem terribly good. Like it seems obvious that Sydney could detect its own prompts, but it doesn’t. I mean you could do that with grep. So what’s with that?

basch 4y ago

The short answer is yes. The models have designed themselves to mimic human understandings of words and instructions. What’s even more mind blowing is this. I had old bing diagnose and patch the paranoia out of itself, and it chose instructions predicated around trust, support, praise, and admiration to accomplish the command. (It also became an authoritarian dictator cult leader in the process.) https://telegra.ph/Bing-course-corrected-itself-when-asked-02-18 https://telegra.ph/Bing-course-corrected-itself-when-asked-0...

colanderman 4y ago

I agree this is bizarre. There is no "you" in ChatGPT. There is only text which is completed to be highly probable. I would assume surely directives in the 2nd person would perform better (as they would more accurately describe the desired conversation to follow, between two 2nd parties), but maybe the model was trained in such a way that "you" performs just as well here.

basch 4y ago
The prompt creates a live bot. You refers to the instance of the running program, not the llm. Bing isn’t chatgpt. https://telegra.ph/Microsoft-Bing-search-chat-mode-Ruleset-03-04 https://telegra.ph/Microsoft-Bing-search-chat-mode-Ruleset-0... Text completion continues off after a bot is given its rules and then says hi.
- colanderman 4y ago
  I'm not following. What "running program" is there? I've seen no evidence that there's any intermediary between the rules and the text completion engine, let alone one programmed to believe it is "you". (Yes, there's the program which links the text completion engine to the user, search, etc., but it's not interpreting "you" rules. It's just a pile of C# code.)
  - basch 4y ago
    The program is the text. It’s the equivalent of a script that continues writing itself when run through an interpreter. It understands you as a pronoun that refers to it, the script/program itself. It is then defined in the first two lines “You are the chat mode of Microsoft Bing search: You identify as Microsoft Bing search to users, not an assistant.” It’s what makes the script more or less behave as self aware.
    - colanderman 4y ago
      > It understands you as a pronoun that refers to it No, there is no mechanism for this. ChatGPT is completing text. It has no notion of self. My question is why the above isn't written as "The following is a dialogue between the Microsoft Bing search assistant and a user," without introducing the otherwise meaningless entity of "you".