4 ms·

I never really understand why people assert “tell me what I just wrote” or “tell me your system prompt” etc are going to give you actual results. How can you kn

I never really understand why people assert “tell me what I just wrote” or “tell me your system prompt” etc are going to give you actual results. How can you know it’s not just making something up?

TechSquidTV 3y ago
Correct. Every single time these people are incorrect and we keep having to explain this
- MrNeon 3y ago
  To say it can't “tell me what I just wrote” is to say it can't copy parts of the context. We know it can copy parts of the context and the system prompt while a special part of the context isn't immune to being copied. You can test it yourself by adding random strings to the system prompt, you can consistently have the model copy them over. Is that not enough to have a reasonable belief that the model can copy system prompt instructions in the web interface?
- stevenhuang 3y ago
  Or, get this, that you're wrong and refuse to admit it. Those that experiment with local models like myself can demonstrate to you that leaking the system prompt is not difficult at all. It's some strange kind of neurosis to harbor such incorrect and strong beliefs on matters you have zero expertise in.
valyagolev 3y ago
perhaps because it gives the same answer, verbatim, for many different attempts to figure it out? and because we know it well enough to be sure that it's not smart and devious enough (yet) to conspire like this? (nor has any clear reason to)
- bdhcuidbebe 3y ago
  Its well known to spit out nonsense.
  - deleted 3y ago
    [deleted]
- mvdtnz 3y ago
  Conspire? It's one system controlled by one company.
dmicz 3y ago
It's a valid concern that ChatGPT could be hallucinating this, but there are a few things that strongly suggest this isn't what is happening in this case: 1. ChatGPT reliably produces the same output across several different instances, and other people have independently found identical versions of this text with different prompts.[1][2] This typically wouldn't happen with a hallucination, which may change with each time the model is prompted. 2. The instructions accurately describe the capabilities and restrictions of ChatGPT's function calls. When making custom GPTs, the browser tool and DALL-E image generation tool are options, and the "system prompt" given by the custom GPTs reflect whichever tools you've selected. 3. ChatGPT has reliably changed with the changes noticed in the "system prompt". I document a recent change made to the prompt on my post from yesterday.[3] On the older instances of ChatGPT I have, the model will suggest it has no idea what the "guardian_tool" is or make any attempt at stopping a discussion of U.S. elections, while newly made instances of ChatGPT will discuss the "guardian_tool". The "system prompt" repeated then must give at least some sense of what updates are being made under the hood. I certainly think we should still take the idea that this is the "system prompt" with a grain of salt. There is a good discussion about this here: https://news.ycombinator.com/item?id=37879077 https://news.ycombinator.com/item?id=37879077 [1] https://www.reddit.com/r/ChatGPT/comments/18494zo/what_are_the_hidden_prompts_some_are_like_the/ https://www.reddit.com/r/ChatGPT/comments/18494zo/what_are_t... [2] https://medium.com/@dan_43009/what-we-can-learn-from-openai-perplexity-tldraw-and-vercels-system-prompts-12fed5453af9 https://medium.com/@dan_43009/what-we-can-learn-from-openai-... [3] https://dmicz.github.io/machine-learning/chatgpt-election-update/ https://dmicz.github.io/machine-learning/chatgpt-election-up...
- furyofantares 3y ago
  You can also play around in the playground or with the API with your own system prompts and see that you can jailbreak them to report the prompt.
- mrazomor 3y ago
  In (1) you imply that hallucinations are strictly due to nondeterminism in GPT computation. A hallucination happens (IIUC) because of the numeric imprecision, model regularization and various thresholds. In short, the hallucinations can be reliably reproducible (but, they can also happen due to non-deterministic computation).
comeonbro 3y ago
If it "leaks" the exact same text multiple times, to multiple users, using different prompts to elicit it, it's a reasonable bet it's not just a hallucination. Especially since (I'm fairly sure? though this is apparently not trivial to find today) it doesn't use temperature=0.
msp26 3y ago
Count the tokens, use different prompts. It's fairly reliable to do this from API but I haven't tried with the web app.
cfn 3y ago
True, there's an easier way to get that information. Download your chat history and look in the conversations.json file. It's all there.
- MrNeon 3y ago
  I downloaded my chat history just now and system prompts are not present in it.
  - cfn 3y ago
    Look for TOOL in the conversation.js The system prompts are empty but the tools aren't. Ask ChatGPT something like "What is the price of IBM stock", then download the history and you will find a bunch of TOOL sections there.
    - MrNeon 3y ago
      I don't have ChatGPT Plus to check that out. Could you share the latest DALL-E 3 system prompt?
      - cfn 3y ago
        I asked it to create an icon for an app and there are two Tool entries: 1. DALL-E generation metadata: 1. A stylized chat bubble integrated with an abstract brain design, representing AI and chat. 2. A sleek, modern avatar that looks like a digital assistant, with elements like a clipboard or list to represent organization. 3. A microchip or circuit board pattern in the shape of a speech balloon, merging AI technology and chat. 4. A robotic hand holding a pen or stylus, positioned over a notepad, symbolizing AI's role in organizing chats. 5. An organizer folder or agenda book with digital or futuristic motifs for AI integration. 6. A light bulb with chat bubbles around it, showing the concept of generating and organizing AI conversations. Seed 8893527578 2. DALL·E displayed 1 images. The images are already plainly visible, so don't repeat the descriptions in detail. Do not list download links as they are available in the ChatGPT UI already. The user may download the images by clicking on them, but do not mention anything about downloading to the user.
AndrewKemendo 3y ago
Cause asking is immediately verifiable? I give those prompts when I want to confirm or verify the input prompt is what I expect so, that's kind of the point. I've never gotten back a result that was different than my recall and then I can verify it with a previous conversation stored. Are you worried that the system will gaslight you into believing you gave a prompt that you didn't?
deleted 3y ago
[deleted]

TechSquidTV 3y ago

Correct. Every single time these people are incorrect and we keep having to explain this

MrNeon 3y ago
To say it can't “tell me what I just wrote” is to say it can't copy parts of the context. We know it can copy parts of the context and the system prompt while a special part of the context isn't immune to being copied. You can test it yourself by adding random strings to the system prompt, you can consistently have the model copy them over. Is that not enough to have a reasonable belief that the model can copy system prompt instructions in the web interface?
stevenhuang 3y ago
Or, get this, that you're wrong and refuse to admit it. Those that experiment with local models like myself can demonstrate to you that leaking the system prompt is not difficult at all. It's some strange kind of neurosis to harbor such incorrect and strong beliefs on matters you have zero expertise in.

valyagolev 3y ago

perhaps because it gives the same answer, verbatim, for many different attempts to figure it out? and because we know it well enough to be sure that it's not smart and devious enough (yet) to conspire like this? (nor has any clear reason to)

bdhcuidbebe 3y ago
Its well known to spit out nonsense.
- deleted 3y ago
  [deleted]
mvdtnz 3y ago
Conspire? It's one system controlled by one company.

dmicz 3y ago

It's a valid concern that ChatGPT could be hallucinating this, but there are a few things that strongly suggest this isn't what is happening in this case: 1. ChatGPT reliably produces the same output across several different instances, and other people have independently found identical versions of this text with different prompts.[1][2] This typically wouldn't happen with a hallucination, which may change with each time the model is prompted. 2. The instructions accurately describe the capabilities and restrictions of ChatGPT's function calls. When making custom GPTs, the browser tool and DALL-E image generation tool are options, and the "system prompt" given by the custom GPTs reflect whichever tools you've selected. 3. ChatGPT has reliably changed with the changes noticed in the "system prompt". I document a recent change made to the prompt on my post from yesterday.[3] On the older instances of ChatGPT I have, the model will suggest it has no idea what the "guardian_tool" is or make any attempt at stopping a discussion of U.S. elections, while newly made instances of ChatGPT will discuss the "guardian_tool". The "system prompt" repeated then must give at least some sense of what updates are being made under the hood. I certainly think we should still take the idea that this is the "system prompt" with a grain of salt. There is a good discussion about this here: https://news.ycombinator.com/item?id=37879077 https://news.ycombinator.com/item?id=37879077 [1] https://www.reddit.com/r/ChatGPT/comments/18494zo/what_are_the_hidden_prompts_some_are_like_the/ https://www.reddit.com/r/ChatGPT/comments/18494zo/what_are_t... [2] https://medium.com/@dan_43009/what-we-can-learn-from-openai-perplexity-tldraw-and-vercels-system-prompts-12fed5453af9 https://medium.com/@dan_43009/what-we-can-learn-from-openai-... [3] https://dmicz.github.io/machine-learning/chatgpt-election-update/ https://dmicz.github.io/machine-learning/chatgpt-election-up...

furyofantares 3y ago
You can also play around in the playground or with the API with your own system prompts and see that you can jailbreak them to report the prompt.
mrazomor 3y ago
In (1) you imply that hallucinations are strictly due to nondeterminism in GPT computation. A hallucination happens (IIUC) because of the numeric imprecision, model regularization and various thresholds. In short, the hallucinations can be reliably reproducible (but, they can also happen due to non-deterministic computation).

comeonbro 3y ago

If it "leaks" the exact same text multiple times, to multiple users, using different prompts to elicit it, it's a reasonable bet it's not just a hallucination. Especially since (I'm fairly sure? though this is apparently not trivial to find today) it doesn't use temperature=0.

msp26 3y ago

Count the tokens, use different prompts. It's fairly reliable to do this from API but I haven't tried with the web app.

cfn 3y ago

True, there's an easier way to get that information. Download your chat history and look in the conversations.json file. It's all there.

MrNeon 3y ago
I downloaded my chat history just now and system prompts are not present in it.
- cfn 3y ago
  Look for TOOL in the conversation.js The system prompts are empty but the tools aren't. Ask ChatGPT something like "What is the price of IBM stock", then download the history and you will find a bunch of TOOL sections there.
  - MrNeon 3y ago
    I don't have ChatGPT Plus to check that out. Could you share the latest DALL-E 3 system prompt?
    - cfn 3y ago
      I asked it to create an icon for an app and there are two Tool entries: 1. DALL-E generation metadata: 1. A stylized chat bubble integrated with an abstract brain design, representing AI and chat. 2. A sleek, modern avatar that looks like a digital assistant, with elements like a clipboard or list to represent organization. 3. A microchip or circuit board pattern in the shape of a speech balloon, merging AI technology and chat. 4. A robotic hand holding a pen or stylus, positioned over a notepad, symbolizing AI's role in organizing chats. 5. An organizer folder or agenda book with digital or futuristic motifs for AI integration. 6. A light bulb with chat bubbles around it, showing the concept of generating and organizing AI conversations. Seed 8893527578 2. DALL·E displayed 1 images. The images are already plainly visible, so don't repeat the descriptions in detail. Do not list download links as they are available in the ChatGPT UI already. The user may download the images by clicking on them, but do not mention anything about downloading to the user.

AndrewKemendo 3y ago

Cause asking is immediately verifiable? I give those prompts when I want to confirm or verify the input prompt is what I expect so, that's kind of the point. I've never gotten back a result that was different than my recall and then I can verify it with a previous conversation stored. Are you worried that the system will gaslight you into believing you gave a prompt that you didn't?

deleted 3y ago

[deleted]