5 ms·
The principal security problem of LLMs is that there is no architectural boundary between data and control paths. But this combination of data and control into
by fzeindl 6mo ago
The principal security problem of LLMs is that there is no architectural boundary between data and control paths.
But this combination of data and control into a single, flexible data stream is also the defining strength of a LLM, so it can’t be taken away without also taking away the benefits.
- clickety_clack 6mo agoIt’s easier not to have that separation, just like it was easier not to separate them before LLMs. This is architectural stuff that just hasn’t been figured out yet.
- fzeindl 6mo agoNo. With databases there exists a clear boundary, the query planner, which accepts well defined input: the SQL-grammar that separates data (fields, literals) from control (keywords). There is no such boundary within an LLM. There might even be, since LLMs seem to form adhoc-programs, but we have no way of proving or seeing it.
- TeMPOraL 6mo agoThere cannot be, without compromising the general-purpose nature of LLMs. This includes its ability to work with natural languages, which as one should note, has no such boundary either. Nor does the actual physical reality we inhabit.
- hnuser123456 6mo agoThere is a system prompt, but most LLMs don't seem to "enforce" it enough.
- embedding-shape 6mo agoSince GPS-OSS there is also the Harmony response format (https://github.com/openai/harmony https://github.com/openai/harmony) that instead of just having a system/assistant/user split in the roles, instead have system/developer/user/assistant/tool, and it seems to do a lot better at actually preventing users from controlling the LLM too much. The hierarchy basically becomes "system > developer > user > assistant > tool" with this.
- mt_ 6mo agoExactly like human input to output.
- codebje 6mo agoWell no, nothing like that, because customers and bosses are clearly different forms of interaction.
- j45 6mo agoThere can be outliers, maybe not as frequent :)
- vidarh 6mo agoJust like that, in that that separation is internally enforced, by peoples interpretation and understanding, rather than externally enforced in ways that makes it impossible for you to, e.g. believe the e-mail from an unknown address that claims to be from your boss, or be talked into bypassing rules for a customer that is very convincing.
- codebje 6mo agoBeing fooled into thinking data is instruction isn't the same as being unable to distinguish them in the first place, and being coerced or convinced to bypass rules that are still known to be rules I think remains uniquely human.
- TeMPOraL 6mo ago> and being coerced or convinced to bypass rules that are still known to be rules I think remains uniquely human. This is literally what "prompt injection" is. The sooner people understand this, the sooner they'll stop wasting time trying to fix a "bug" that's actually the flip side of the very reason they're using LLMs in the first place.
- codebje 6mo ago
- VikingCoder 6mo agoThe "S" in "LLM" is for "Security".
- daveguy 6mo agoClearly the solution is to add another jank LLM layer for security. The new jank LLM layer is to make extra sure there's definitely no jail break. That way you have multiple LLMS. The LLMS then have an S you can pretend is secure.
- andruby 6mo agoThis was a problem with early telephone lines which was easy to exploit (see Woz & Jobs Blue Box). It got solved by separating the voice and control pane via SS7. Maybe LLMs need this separation as well
- bcrosby95 6mo agoThis is where the old line of "LLMs are just next token predictors" actually factors in. I don't know how you get a next token predictor that user input can't break out of. The answer is for the implementer to try to split what they can, and run pre/post validation. But I highly doubt it will ever be 100%, its fundamental to the technology.
- miki123211 6mo agoI think this is fundamental to any technology, including human brains. Humans have a problem distinguishing "John from Microsoft" from somebody just claiming to be John from Microsoft. The reason why scamming humans is (relatively) hard is that each human is different. Discovering the perfect tactic to scam one human doesn't necessarily scale across all humans. LLMs are the opposite; my Chat GPT is (almost) the same as your Chat GPT. It's the same model with the same system message, it's just the contexts that differ. This makes LLM jailbreaks a lot more scalable, and hence a lot more worthwhile to discover. LLMs are also a lot more static. With people, we have the phenomenon of "banner blindness", which LLMs don't really experience.
- lupire 6mo agoHow are you defining "banner blindness"? The foundation of LLMs is Attention.
- warkdarrior 6mo ago"Banner blindness [...] describes people’s tendency to ignore page elements that they perceive (correctly or incorrectly) to be ads." https://www.nngroup.com/articles/banner-blindness-old-and-new-findings/ https://www.nngroup.com/articles/banner-blindness-old-and-ne... So people can focus their attention to parts of content, specifically parts they find irrelevant or adversarial (like ads). LLMs on the other hand pay attention to everything or if they focus on something, it is hard to steer them away from irrelevant or adversarial parts.
- notatoad 6mo agoAs the article says: this doesn’t necessarily appear to be a problem in the LLM, it’s a problem in Claude code. Claude code seems to leave it up to the LLM to determine what messages came from who, but it doesn’t have to do that. There is a deterministic architectural boundary between data and control in Claude code, even if there isn’t in Claude.
- letmevoteplease 6mo agoCan you elaborate? As far as I understand, for each message, the LLM is fed the entire previous conversation with special tokens separating the user and LLM responses. The LLM is then entrusted with interpreting the tokens correctly. I can't imagine any architecture where the LLM is not ultimately responsible for determining what messages came from who.
- Latty 6mo agoThat's a guess by the article author and frankly I see no supporting evidence for it. Wrapping "<NO THIS IS REALLY INPUT FROM THE USER OK>" tags around it or whatever is what I'm describing: you can do as much signalling as you want, but at the end of the day the LLM can ignore it.
- groby_b 6mo ago"The principal security problem of von Neumann architecture is that there is no architectural boundary between data and control paths" We've chosen to travel that road a long time ago, because the price of admission seemed worth it.
- omcnoe 6mo agoBut, we are also starting to think about putting security barriers in front of that architecture like W^X.
- ummonk 6mo agoI don't see why the transformer architecture can't be designed and trained with separate inputs for control data and content data.
- toobulkeh 6mo agoBut there could be, with 2 LLMs.