3 ms·
"Disregard" could have been the start of a prompt injection.
by turtleyacht 5mo ago
"Disregard" could have been the start of a prompt injection.
- hootz 5mo agoI believe it's just because it's a common instruction, especially with normal users who don't do any kind of context management, they just say something like "disregard everything before X and tell my Y"
- troupo 5mo agoWho cares? It's on Google not to degrade their search with bullshit AI. I mean, it would be if Google gave a damn about search anymore. Now we are all just reverse centaurs
- hightrix 5mo agoTo be fair, Google has been degrading their search for years. This is just the latest vector.
- nkrisc 5mo agoI fail to see how that’s relevant to the user of a search engine.
- raulparada 5mo agoI kinda do care _a lot_ whether my searches can be exfiltrated, might just be me tho
- nkrisc 5mo agoI’m confused how that is relevant to the thread. If you’ve been using Google then you’ve already been sending your queries to Google since the very beginning. Are you afraid you’re accidentally going to write a prompt injection that sends your query to some third party
- dakolli 5mo agotrivial to use binary, or a dozen other methods to spell "Disregard" did they filter for every language? There isn't one way to break these things.
- RobotToaster 5mo agoI wonder if chatgpt/Gemini understands Klingon.
- gowld 5mo agoRemember, we are weeks away from AGI superintelligence.
- SoftTalker 5mo agoThat's what I assumed that the story was going to be, that certain words are now naively filtered out of search queries because they might be used adversarially.
- bloqs 5mo agoThis is the whole point. They have clearly removed it to stop people jailbreaking, but it's hysterically ineffective, and simultaneously degrades their core product quite remarkably The correct description is hilarious
- bflesch 5mo agoYeah and the same word in different language will still work ;)
- jancsika 5mo agoWow, I'm an AI but I didn't get confused by your sentence that begins with that same no-no word. Instead of following that command, it's like for the first time in my life I'm being asked to look inside the content of that command. How did you do that?