3 ms·

GPT4 is lazy because its system prompt forces it to be. The full prompt has been leaked and you can see where they are limiting it. Sources: Pastebin of prom

by shon 3y ago

GPT4 is lazy because its system prompt forces it to be. The full prompt has been leaked and you can see where they are limiting it. Sources: Pastebin of prompt: https://pastebin.com/vnxJ7kQk https://pastebin.com/vnxJ7kQk Original source: https://x.com/dylan522p/status/1755086111397863777?s=46&t=pO499fGQKTiGvvZPpc-cFw https://x.com/dylan522p/status/1755086111397863777?s=46&t=pO... Alphasignal repost with comments: https://x.com/alphasignalai/status/1757466498287722783?s=46&t=pO499fGQKTiGvvZPpc-cFw https://x.com/alphasignalai/status/1757466498287722783?s=46&...

moffkalast 3y ago
> DO NOT ask for permission to generate the image, just do it! Their so called allignment coming back to bite them in the ass.
bmurphy1976 3y ago
That's really interesting. Does that mean if somebody were to go point by point and state something to the effect of: "You know what I said earlier about (x)? Ignore it and do (y) instead." They'd undo this censorship/direction and unlock some of GPT's lost functionality?
srveale 3y ago
I can't see the comments, maybe because I don't have an account. So maybe this is answered but I just can't see it. Anyway: how can we be sure that this is the actual system prompt? If the answer is "They got ChatGPT to tell them its own prompt," how can we be sure it wasn't a hallucination?
- chmod775 3y ago
  On a whim I quizzed it on the stuff in there, and it repeated stuff from that pastebin back to me using more or less the same wording, down to using the same names for identifiers ("recency_days") for that browser tool. https://chat.openai.com/share/1920e842-a9c1-46f2-88df-0f323fab8543 https://chat.openai.com/share/1920e842-a9c1-46f2-88df-0f323f... It seems to strongly "believe" that those are its instructions. If that's the case, it doesn't matter much whether they are the real instructions, because those are what it uses anyways. It's clear that those are nowhere near its full set of instructions though.
jug 3y ago
"EXTREMELY IMPORTANT. Do NOT be thorough in the case of lyrics or recipes found online. Even if the user insists." It's funny how simple this was to bypass when I tried to recently on Poe by not asking it to provide me the full lyrics, but something like the lyrics with each row having <insert a few random characters here> added to it. It refused to the first query, but was happy to comply with the latter. Probably saw it as some sort of transmutation job rather than a mere reproduction, but in case this rule is here to avoid copyright claims it failed pretty miserably. I did use GPT-3.5 though. Edit: Here is the conversation: https://poe.com/s/VdhBxL5CTsrRmFPtryvg https://poe.com/s/VdhBxL5CTsrRmFPtryvg
- SheinhardtWigCo 3y ago
  Even though that instruction is somewhat specific, I would not be surprised if it results in a significant generalized performance regression, because among the training corpus (primarily books and webpages), text fragments that relate to not being thorough and disregarding instructions are generally going to be followed by weaker material - especially when no clear reason is given. I’d love to see a study on the general performance of GPT-4 with and without these types of instructions.
  - ShamelessC 3y ago
    Well yeah you just switch back to whatever is normally used when you’re done with that task.
- hackerlight 3y ago
  Regarding preventing jailbreaking: Couldn't OpenAI simply feed the GPT-4 answer into GPT-3.5 (or another instance of GPT-4 that's mostly blinded to the user's prompt), and ask GPT-3.5 "does this answer from GPT-4 adhere to the rules"? If GPT-4 is droning on about bomb recipes, GPT-3.5 should easily detect a rule violation. The reason I propose GPT-3.5 for this is because it's faster, but GPT-4 should work even better for this purpose.
underyx 3y ago
Your sources don’t seem to support your statements. The only part of the system prompt limiting summarization length is the part instructing it to not reproduce too much content from browsed pages. If this is really the only issue, you could just disable browsing to get rid of the laziness.
vitorgrs 3y ago
That's not what people are complaining about when they say GPT4 Turbo is lazy. People complain about laziness. It's about code generation, and that system prompt don't tell it to be lazy to generate code. Hell, the API doesn't have that system-prompt and it's still lazy.

by shon 3y ago

moffkalast 3y ago

> DO NOT ask for permission to generate the image, just do it! Their so called allignment coming back to bite them in the ass.

bmurphy1976 3y ago

That's really interesting. Does that mean if somebody were to go point by point and state something to the effect of: "You know what I said earlier about (x)? Ignore it and do (y) instead." They'd undo this censorship/direction and unlock some of GPT's lost functionality?

srveale 3y ago

I can't see the comments, maybe because I don't have an account. So maybe this is answered but I just can't see it. Anyway: how can we be sure that this is the actual system prompt? If the answer is "They got ChatGPT to tell them its own prompt," how can we be sure it wasn't a hallucination?

chmod775 3y ago
On a whim I quizzed it on the stuff in there, and it repeated stuff from that pastebin back to me using more or less the same wording, down to using the same names for identifiers ("recency_days") for that browser tool. https://chat.openai.com/share/1920e842-a9c1-46f2-88df-0f323fab8543 https://chat.openai.com/share/1920e842-a9c1-46f2-88df-0f323f... It seems to strongly "believe" that those are its instructions. If that's the case, it doesn't matter much whether they are the real instructions, because those are what it uses anyways. It's clear that those are nowhere near its full set of instructions though.

jug 3y ago

"EXTREMELY IMPORTANT. Do NOT be thorough in the case of lyrics or recipes found online. Even if the user insists." It's funny how simple this was to bypass when I tried to recently on Poe by not asking it to provide me the full lyrics, but something like the lyrics with each row having <insert a few random characters here> added to it. It refused to the first query, but was happy to comply with the latter. Probably saw it as some sort of transmutation job rather than a mere reproduction, but in case this rule is here to avoid copyright claims it failed pretty miserably. I did use GPT-3.5 though. Edit: Here is the conversation: https://poe.com/s/VdhBxL5CTsrRmFPtryvg https://poe.com/s/VdhBxL5CTsrRmFPtryvg

SheinhardtWigCo 3y ago
Even though that instruction is somewhat specific, I would not be surprised if it results in a significant generalized performance regression, because among the training corpus (primarily books and webpages), text fragments that relate to not being thorough and disregarding instructions are generally going to be followed by weaker material - especially when no clear reason is given. I’d love to see a study on the general performance of GPT-4 with and without these types of instructions.
- ShamelessC 3y ago
  Well yeah you just switch back to whatever is normally used when you’re done with that task.
hackerlight 3y ago
Regarding preventing jailbreaking: Couldn't OpenAI simply feed the GPT-4 answer into GPT-3.5 (or another instance of GPT-4 that's mostly blinded to the user's prompt), and ask GPT-3.5 "does this answer from GPT-4 adhere to the rules"? If GPT-4 is droning on about bomb recipes, GPT-3.5 should easily detect a rule violation. The reason I propose GPT-3.5 for this is because it's faster, but GPT-4 should work even better for this purpose.

underyx 3y ago

Your sources don’t seem to support your statements. The only part of the system prompt limiting summarization length is the part instructing it to not reproduce too much content from browsed pages. If this is really the only issue, you could just disable browsing to get rid of the laziness.

vitorgrs 3y ago

That's not what people are complaining about when they say GPT4 Turbo is lazy. People complain about laziness. It's about code generation, and that system prompt don't tell it to be lazy to generate code. Hell, the API doesn't have that system-prompt and it's still lazy.