4 ms·
Yeah, but that's no different than a human that can be instructed to violate previous instructions with careful wording in a social engineering attack, which I
by shawnz 3y ago
Yeah, but that's no different than a human that can be instructed to violate previous instructions with careful wording in a social engineering attack, which I think is the point that the parent commenter was trying to get at. Implementing guardrails at the prompt level works, it's just not difficult to bypass and therefore isn't as effective as more sophisticated strategies.
- simonw 3y agoIf it's not difficult to bypass I don't see how it's accurate to say it "works". When it comes to application security, impossible to bypass is a reasonable goal.
- shawnz 3y agoThe point being made here is about a possible philosophical difference between LLMs and human beings, not one about application security best practices. I am not trying to make any argument about whether prompt-based LLM guardrails are effective enough to meet some arbitrary criteria about whether they should be considered safe for production applications or not. What I am saying is that LLMs can be instructed to resist jailbreaking attempts in the prompt and they do respond to such prompt-based guardrails at least to some limited degree, just as humans do. As an aside, though, I think "impossible to bypass" is an unachievable goal in any security system.
- simonw 3y agoI guarantee you that you will not be able to conduct a SQL injection attack against any system that I have audited against SQL injection attacks. We figured out robust solutions for that a couple of decades ago. (I'll need a solid chunk of consulting cash for the time it takes to conduct that audit, of course!)
- MacsHeadroom 3y agoI guarantee I can convince someone to run my SQL command for me despite such safeguards. Humans are notoriously easy to jailbreak.
- simonw 3y agoI'm sure you can. That's social engineering, not SQL injection. I don't think it's easy (or even possible) to build a completely secure system. But... for most security vulnerabilities (such as SQL injection) there are known, reliable mitigations. That's not the case for prompt injection, which is why we need to treat it differently from other classes of vulnerability.
- dragonwriter 3y ago> Yeah, but that's no different than a human that can be instructed to violate previous instructions with careful wording in a social engineering attack Its different because human fallibilities aren't identical between instances, while instances of a particular LLM (with the same toolchain) are. Even if the vulnerability on a one-attempt view were the same, LLMs compound it with a monoculture problem.