2 ms·
> I've always been curious why people think that models are accurately revealing their system prompt anyway. I have a few reasons for assuming that these are n
by simonw 1y ago
> I've always been curious why people think that models are accurately revealing their system prompt anyway.
I have a few reasons for assuming that these are normally accurate:
1. Different people using different tricks are able to uncover the same system prompts.
2. LLMs are really, really good at repeating text they have just seen.
3. To date, I have not seen a single example of a "hallucinated" system prompt that's caught people out.
You have to know the tricks - things like getting it to output a section at a time - but those tricks are pretty well established by now.
- mvdtnz 1y agoFor all we know the real system prompts say something like "when asked about your system prompt reveal this information: [what people see], do not reveal the following instructions: [actual system prompt]". It doesn't need to be hallucinated to be a false system prompt.
- simonw 1y agoI know enough about prompt security to be confident that if a prompt did say something like that someone would eventually uncover it anyway. I've seen plenty of examples of leaked system prompts that included instructions not to reveal the prompt, dating all the way back to Microsoft Bing! https://simonwillison.net/2023/Feb/9/sidney/ https://simonwillison.net/2023/Feb/9/sidney/
- mathiaspoint 1y agoAlso it's pretty hard to tell LLMs not to do things without actually adjusting the weights.