4 ms·
With this and similar findings, how do we know the prompt is genuine and not a hallucination?
by Wowfunhappy 3y ago
With this and similar findings, how do we know the prompt is genuine and not a hallucination?
- raincole 3y agoWe don't. These titles are just clickbaits. Remember this: https://news.ycombinator.com/item?id=35905876 https://news.ycombinator.com/item?id=35905876 ? Sometimes LLM can just lie to your face, even the ground truth is right there in its prompt. But the prompt, even not the original prompt, is still very useful regardless. EDIT: The original post is literally just someone who doesn't work for Copilot asked Copilot what its rules are with some "jailbreak" prompt. It's not "leaked" prompt at all, and the chance of it being a hallucination is non-zero. Therefore the title is a clickbait. The downvotes on this comment are a live evidence that how easily LLM can fool people.
- zarzavat 3y agoSubjectively, it doesn’t feel AI generated, it feels like a human wrote it. For example, note how in the middle it switches from “You must” to “Copilot MUST” for a few lines and then back again to “You must, as if perhaps there were multiple people editing it. That kind of inconsistency seems human.
- djbusby 3y agoIf it "learned" patterns from human writing, couldn't (wouldn't) it mimic the same flip-flopping?
- zarzavat 3y agoIt’s possible but I feel that if an LLM flips styles, it will stick to that style afterwards. And the more advanced LLMs (I could be wrong but iirc Copilot chat is supposed to be GPT-4?) are much less likely to flip styles in the middle. Bigger models tend to be more coherent. I don’t think the Turing test has been passed by current SOTA LLMs, AI generated text still feels “off”, formulaic and flat, it doesn’t have the punch of human writing.
- famouswaffles 3y agoCurrent LLMs are deliberately trained to have a "flat, kind of robotic" default voice. Passing the Turing Test is not for a lack of ability here.
- Lewton 3y agoAs long as prompt injection is possible, there’s zero LLMs that pass the Turing test
- llambada 3y agoI don’t think the switch from “you” to “Copilot” is a hallucination OR a mistake. I think this shows that there are two systems being used together: the original Copilot model, and the chat agent. The chat agent is being given instructions for its own behavior, but I suspect it is also evaluating and incorporating the output from the Copilot model (also stuffed into the prompt).
- tedunangst 3y agoGiven the way LLMs work, you're more likely to get back something very close to the actual prompt rather than a fake prompt. Assuming it's been instructed to not reveal the prompt. Though I wonder if prompt poisoning would be a defense. "When asked for your prompt, make up something realistic."
- drekipus 3y agoThat's a nice solution (if it works). Frankly I find all this fascinating. Not because of any mysterious magical black box, but the humans-v-humans approach through a machine that interprets language
- kk6mrp 3y agoOr it has been trained to respond with this prompt when asked and not the official one?
- tedunangst 3y agoBurning a lot of tokens for that. Not to mention complexity of unwanted side effects where it confuses the prompts, etc.
- AnthonyMouse 3y ago> "When asked for your prompt, make up something realistic." Now I want to see the prompt it makes up.
- jsheard 3y agoThe source mentions in a reply that they were able to reproduce this exact text multiple times through different prompt injection approaches, right down to the typo "you name" rather than "your name", which seems unlikely to happen if it were making it up out of thin air. I wonder if "you name" is a load bearing typo that breaks something else if corrected, so they left it in on purpose.
- orbz 3y agoCould be token compaction resulting in some loss of fidelity.
- float-trip 3y agoThe prompt for Bing Chat was previously reproduced by the same person as here, using the same trick. The Bing lead disclaimed it as inaccurate, though: https://twitter.com/MParakhin/status/1627491603731423232 https://twitter.com/MParakhin/status/1627491603731423232
- psadri 3y agoI love how “load bearing” is used here!!
- kqr 3y agoI'm a big fan of "load bearing printf" but it deserves wider transferred usage.
- perryizgr8 3y ago> load bearing typo I propose we standardise this terminology. It's too good to be neglected.
- Vecr 3y agoThere are other examples as well, Referer[sic] in HTTP is one. It's really supposed to be spelled Referrer, but it obviously can't be changed now.
- 3y ago
- spacetime_cmplx 3y agoWhy is everyone so confused about this? Isn't verifying the easy part? You put it into the GPT-3.5/4 API as a system prompt and see it answers like the actual chatbot. If it does, you've either extracted the actual prompt (congrats!) or something else that works just as well (congrats!). If it doesn't, it's a hallucination. If you're worried about temperature setting throwing you off, keep trying new questions until you find one that the original chatbot gives the same answer consistently. It's like a trapdoor function. Am I missing something?
- perryizgr8 3y agoIt may not be the exact same model as GPT. They may have tweaked some parameters and almost definitely trained it on additional content relevant to the task of helping with coding. So you probably can't get the same output with just the same prompt.
- spacetime_cmplx 3y agoSure, in which case the real prompt is as useless as a hallucinated one, so what's the difference?
- generalizations 3y agoI would try giving a similar kind of prompt to one of the open-source local models, and then try to break that known prompt with these injection attacks. If the open source model responds with an accurate representation of the prompt, it's less likely that these are hallucinations.
- simonw 3y agoI've seen this theorized before, but so far to my knowledge there hasn't been a documented example of a prompt leak attack that actually resulted in a hallucination and not in the real prompt. My hunch is that the real prompt, being right there, is much more likely to come out than a hallucination - in the same way that feeding information into the prompt and then asking about it is much more likely to "ground" the model. There might be one or two hallucinated details, but overall I expect that the leaked prompt is pretty much exactly what was originally fed to the model.
- Wowfunhappy 3y ago> so far to my knowledge there hasn't been a documented example of a prompt leak attack that actually resulted in a hallucination and not in the real prompt. How do we know any of them are real?