2 ms·
The difference between humans and LLM systems is that, if you try 1,000 different variations of an attack on a pair of humans, they notice. There are plenty of
by simonw 1y ago
The difference between humans and LLM systems is that, if you try 1,000 different variations of an attack on a pair of humans, they notice.
There are plenty of AI-layer-that-detects-attack mechanisms that will get you to a 99% success rate at preventing attacks.
In application security, 99% is a failing grade. Imagine if we prevented SQL injection with approaches that didn't catch 1% of potential attacks!
- TeMPOraL 1y agoThat's a wrong approach. You can't have 100% security when you add LLMs into the loop, for the exact same reason as when you involve humans. Therefore, you should only include LLMs - or humans - in systems where less than 100% success rate is acceptable, and then stack as many mitigations as it takes (and you can afford) to make the failure rate tolerable. (And, despite what some naive takes on infosec would have us believe, less than 100% security is perfectly acceptable almost everywhere, because that's how it is for everything except computers, and we've learned to deal with it.)
- tptacek 1y agoSure you can. You just design the system to assume the LLM output isn't predictable, come up with invariants you can reason with, and drop all the outputs that don't fit the invariants. You accept up front the idea that a significant chunk of benign outputs will be lossily filtered in order to maintain those invariants. This just isn't that complicated; people are super hung up on the idea that an LLM agent is a loop around a single "LLM session", which is not how real agents work.
- TeMPOraL 1y agoFair. > You just design the system to assume the LLM output isn't predictable, come up with invariants you can reason with, and drop all the outputs that don't fit the invariants. Yes, this is what you do, but it also happens to defeat the whole reason people want to involve LLMs in a system in the first place. People don't seem to get that the security problems are the flip side of the very features they want. That's why I'm in favor of anthropomorphising LLMs in this context - once you view the LLM not as a program, but as a something akin to a naive, inexperienced human, the failure modes become immediately apparent. You can't fix prompt injection like you'd fix SQL injection, for more-less the same reason you can't stop someone from making a bad but allowed choice when they delegate making that choice to an assistant, especially one with questionable intelligence or loyalties.
- ethbr1 1y ago> People don't seem to get that the security problems are the flip side of the very features they want. Everyone who's worked in big tech dev got this the first time their security org told them "No." Some features are just bad security and should never be implemented.
- TeMPOraL 1y agoThat's my point, though. Yes, some features are just bad security, but they nevertheless have to be implemented, because having them is the entire point. Security is a means, not an end - something security teams sometimes forget. The only perfectly secure computing system is an inert rock (preferably one drifting in space, infinitely away from people). Anything more useful than that requires making compromises on security.
- ethbr1 1y agoSome features are literally too radioactive to ever implement. As an example, because in hindsight it's one of the things MS handled really well: UAC (aka Windows sudo). It's convenient for any program running on a system to be able to do anything without a user prompt. In practice, that's a huge vector for abuse, and it turns out that crafting a system of prompting around only the most sensitive actions can be effective. It takes time, but eventually the program ecosystem updates to avoid touching those things in that way (because prompts annoy users), prompt instances decrease, and security is improved because they're rare. Proper feature design is balancing security with functionality, but if push comes to shove security should always win. Insecure, functional systems are worthless, unless the consequences of exploitation are immaterial.
- ants_everywhere 1y agoAI/machine learning has been used in Advanced Threat Protection for ages and LLMs are increasingly being used for advanced security, e.g. https://cloud.google.com/security/ai https://cloud.google.com/security/ai The problem isn't the AI, it's hooking up a yolo coder AI to your production database. I also wouldn't hook up a yolo human coder to my production database, but I got down voted here the other day for saying drops in production databases should be code reviewed, so I may be in the minority :-P
- simonw 1y agoUsing non-deterministic statistical systems to help find security vulnerabilities is fine. Using non-deterministic statistical systems as the only defense against security vulnerabilities is disastrous.
- ants_everywhere 1y agoI don't understand why people get hung up on non-determinism or statistics. But most security people understand that there is no one single defense against vulnerabilities. Disastrous seems like a strong word in my opinion. All of medicine runs on non-deterministic statistical tests and it would be hard to argue they haven't improved human health over the last few centuries. All human intelligence, including military intelligence, is non-deterministic and statistical. It's hard for me to imagine a field of security that relies entirely on complete determinism. I guess the people who try to write blockchains in Haskell. It just seems like the wrong place to put the concern. As far as I can see, having independent statistical scores with confidence measures is an unmitigated good and not something disastrous.
- simonw 1y agoSQL injection and XSS both have fixes that are 100% guaranteed to work against every possible attack. If you make a mistake in applying those fixes, you will have a security hole. When you spot that hole you can close it up and now you are back to 100% protection. You can't get that from defenses that use AI models trained on examples.