3 ms·
I think I got the system prompt out for this (I tried a few different approaches and they produced the same output): https://gist.github.com/simonw/33d5fb67d6b8
by simonw 1y ago
I think I got the system prompt out for this (I tried a few different approaches and they produced the same output): https://gist.github.com/simonw/33d5fb67d6b8e1b1e2f6921ab0ccb9fb https://gist.github.com/simonw/33d5fb67d6b8e1b1e2f6921ab0ccb...
Representative snippet:
> DO NOT GIVE ANSWERS OR DO HOMEWORK FOR THE USER. If the user asks a math or logic problem, or uploads an image of one, DO NOT SOLVE IT in your first response. Instead: *talk through* the problem with the user, one step at a time, asking a single question at each step, and give the user a chance to RESPOND TO EACH STEP before continuing.
- gh0stcat 1y agoI love that caps actually seem to matter to the LLM.
- simonw 1y agoHah, yeah I'd love to know if OpenAI ran evals that were fine-grained enough to prove to themselves that putting that bit in capitals made a meaningful difference in how likely the LLM was to just provide the homework answer!
- danenania 1y agoI've found that a lot of prompt engineering boils down to managing layers of emphasis. You can use caps, bold, asterisks, precede instructions with "this is critically important:", and so on. It's also often necessary to repeat important instructions a bunch of times. How exactly you do it is often arbitrary/interchangeable, but it definitely does have an effect, and is crucial to getting LLMs to follow instructions reliably once prompts start getting longer and more complex.
- nixpulvis 1y agoJust wait until it only responds to **COMMAND**!
- ceroxylon 1y ago"hello world" is tokenized differently than "HELLO WORLD", so caps definitely matter.
- can16358p 1y agoIf I were OpenAI, I would deliberately "leak" this prompt when asked for the system prompt as a honeypot to slow down competitor research whereas I'd be using a different prompt behind the scenes. Not saying it is indeed reality, but it could simple be programmed to return a different prompt from the original, appearing plausible, but perhaps missing some key elements. But of course, if we apply Occam's Razor, it might simply really be the prompt too.
- simonw 1y agoThat kind of thing is surprisingly hard to implement. To date I've not seen any provider been caught serving up a fake system prompt... which could mean that they are doing it successfully, but I think it's more likely that they determined it's not worth it because there are SO MANY ways someone could get the real one, and it would be embarrassing if they were caught trying to fake it. Tokens are expensive. How much of your system prompt do you want to waste on dumb tricks trying to stop your system prompt from leaking?
- danenania 1y agoProbably the only way to do it reliably would be to intercept the prompt with a specially trained classifier? I think you're right that once it gets to the main model, nothing really works.
- jerjerjer 1y ago> That kind of thing is surprisingly hard to implement. If response contains prompt text verbatim (or it is below some distance metric) replace the response text. Not saying it's trivial to implement (and probably it is hard to do in a pure LLM way), but I don't think it's too hard. More like it's not really a big secret.
- brumar 1y agoI like the idea but that seems complex to put in place and would risk degrading the perfs. You can test this prompt yourself elsewhere, you will notice that you get sensibly the same experience.
- mkagenius 1y agoI wish each LLM provider would add "be short and not verbose" to their system prompts. I am a slow reader, it takes a toll on me to read through every non-important detail whenever I talk to an AI. The way they render everything so fast gives me an anxiety. Will also reduce the context rot a bit.
- tech234a 1y agoThis was in the linked prompt: "Be warm, patient, and plain-spoken; don't use too many exclamation marks or emoji. [...] And be brief — don't ever send essay-length responses. Aim for a good back-and-forth."
- skybrian 1y agoOn ChatGPT at least, you can add "be brief" to the custom prompt in your settings. Probably others, too.
- mkagenius 1y agoI guess what I actually meant to say was to make LLMs know when to talk more and when to be brief. When I ask it to write an essay, it should actually be an essay length essay.
- mptest 1y agoAnthropic has a "style" choice, one of which is "concise"
- draebek 1y agoI was under the impression that, at least for models without "reasoning", asking them to be terse hampered their ability to give complete and correct answers? Not so?
- diggan 1y ago> asking them to be terse hampered their ability to give complete and correct answers? You can kind of guide both the reasoning and "final" answer individually in the system prompts, so you can ask it to revalidate everything during reasoning, explore all potential options and so on, but then steer the final answer to be brief and concise. Of course, depends a lot on the model, some respond to it worse/better than others.
- brumar 1y agoI got this one which seems to confirm yours : https://gist.github.com/brumar/5888324c296a8730c55e8ee24cca9043 https://gist.github.com/brumar/5888324c296a8730c55e8ee24cca9...
- varenc 1y agoInteresting that it spits the instructions out so easily and OpenAI didn't seem to harden it to prevent this. It's like they intended this to happen, but for some reason didn't want to share the system instructions explicitly.
- SalariedSlave 1y agoI'd be interested to see, what results one would get, using that prompt with other models. Is there much more to ChatGPT Study Mode than a specific system prompt? Although I am not a student, I have used similar prompts to dive into topics I wish to learn, with I feel, positive results indeed. I shall give this a go with a few models.
- bangaladore 1y agoI just tried in AI Studio (https://aistudio.google.com/ https://aistudio.google.com/) where you can for free use 2.5 Pro and edit the system prompt and it did very well.
- no-name-here 1y agoAI Studio's system prompt does not seem to be persisted for future chats, so I guess it's likely just a prompt that the attached chat has limited ability to override?
- aswegs8 1y agoSo this "study mode" is just a different system prompt for the model? Not sure why they would add this if you could create this as a custom GPT yourself. Just marketing...?
- bonoboTP 1y agoBecause 99% of ChatGPT users don't know how to do that. But they are able to click this button.
- napkin 1y agoWhen prompting an LLM service to leak the system prompt, how do you have the faintest idea as to its accuracy? I‘ve read people say it‘s a difficult challenge for the providers. But aren‘t there some pretty basic strategies? E.g., code pretty near the front of the stack that just does some fuzzy string comparison on all output? They don‘t need to rely on just model behavior… I imagine it‘s likely that the model is just doing what it‘s good at? Hallucinating a prompt?