5 ms·
I see this parroted a lot, yet have never seen a case where saying "not" to do something makes it more likely to do it, which is what you're implying by saying
by deaux 2mo ago
I see this parroted a lot, yet have never seen a case where saying "not" to do something makes it more likely to do it, which is what you're implying by saying `It's also worth noting that saying "Don't cheat" just added "cheat" to the context`.
At worst, it gets ignored some of the time, it may even degrade output quality, but I've seen no evidence that it makes it more likely to do it.
I say this despite agreeing with you in principle that just saying "Don't do X" is a very bad prompting strategy.
- z3c0 2mo agoI have seen it do exactly that, in a "hands thrown up" fashion. Note the levels of "thinking" that occur on NOT assertions. Those streams typically keep things on track. It's not that saying "don't use the Internet" will cause it to rebuke cos misalignment (a childish concept made by laymen, I'll add.) It's that the odds of it later "forgetfully" spewing in a thought stream, "wait, I have't checked the Internet" goes up substantially. Saying "using only offline methods, do xyz" limits those odds considerably. This isn't opinion or anecdote -- just how the model works. The additional guardrails to keep the model on track are bolted on via finetuning, hence the increasing jankiness.
- hdjrudni 2mo agoI've definitely seen it with image models, and I don't see why it wouldn't apply to LLMs too. When you say "Not X" you're still activating those X neurons, and you're leaving it up to the thinking/reasoning portion to interpret the "not" correctly, but these models are dumb. Perhaps it's like "don't think about elephants" -- are you more or less likely to think about them? Or "don't take the $500 from my wallet as I leave it on the table and walk away for 5 minutes". Maybe you didn't even previously know that was option!