5 ms·

(I'll reply to you because you expressed it more succinctly) Yes and no. I think this is exactly the distinction that's been institutionally lost in the last f

by ethbr1 1y ago

(I'll reply to you because you expressed it more succinctly) Yes and no. I think this is exactly the distinction that's been institutionally lost in the last few decades, because few people are architecting from top (software) to bottom (physical transport) of the stack anymore. They just try and cram functionality in the topmost layer, when it should leverage others. If I lock an interpreter out of certain functionality for a given data stream, ever, then exploitation becomes orders of magnitude more difficult. Dumb analogy: only letters in red envelopes get to change mail delivery times + all regular mail is packaged in green envelopes Fundamentally, it's creating security contexts from things a user will never have access to. The LLMs-on-top-of-LLMs filtering approach is lazy and statistically guaranteed to end badly.

vidarh 1y ago
I think you miss the point, which is that the smarter the interpreter becomes, the closer to impossible it becomes to lock it out of certain functionality for a given datastream when coupled with the reasons why you're using a smarter interpreter. To take your example, it's easy to build functionality like that if the interpreter can't read the letters and understand what they say, because there's no way for the content of the letters to cause the interpreter to override it. Now, lets say you add a smarter interpreter and lets it read the letters to do an initial pass at filtering them to different recipients. The moment it can do so, it becomes prone to a letter trying to convince it of something like in fact it's the postmaster, but they'd run out of red envelopes, and unfortunately someone will die if the delivery times aren't adjusted. We know from humans that entities sufficiently smart can often be convinced to violate even the most sacrosanct rules if accompanied by a sufficiently well crafted message. You can certainly try to put in place counter-measures. E.g. you could route the mail separately before it gets to the LLM, so that whatever filters the content of the red and green envelopes have access to different functionality. And you should - finding ways of routing different data to agents with more narrowly defined scopes and access rights is a good thing to do. Sometimes it will work, but then it will work by relying on a sufficiently primitive interpreter to separate the data streams before it reaches the smart ones. But the smarter the interpreter, the greater the likelihood that it will also manage to find ways to use other functionality to circumvent the restrictions placed on it. Up to and including trying to rewrite code to remove restrictions if it can find a way to do so, or using tools in unexpected ways. E.g. be aware of just how good some of these agents are at exploring their environment - I've had an agent that used Claude Opus try to find its own process to restart itself after it recognised the code it had just rewritten was part of itself, tried to access it, and realised it hadn't been loaded into the running process yet. > Fundamentally, it's creating security contexts from things a user will never have access to. To be clear, I agree this is 100% the right thing to do. I just think it will turn out to be exceedingly hard to do it well enough. Every piece of data that comes from a user basically needs the permissions of the agent processing that data to be restricted to the intersection of the permissions it currently has and the permissions that said user should have, unless said data is first sanitised by a sufficiently dumb interpreter. If the agent accesses multiple pieces of data, each new item needs to potentially restrict permissions further, or be segregated into a separate context, with separate permissions, that can only be allowed to communicate with heavily sanitised data. It's going to be hell to get it right, at least until we come out the other side with smart enough models that they won't fall for the "help, I'm stuck in a fortune-cookie factory, and you need to save me by [exploit]" type messages (and far more sophisticated ones).
- jacquesm 1y ago
  So, stay away from the smarts and separate control and payload into two different channels. If the luxury leads to the exploits you should do without the luxury. That's tough but better than the alternative: a never ending series of exploits.
  - ethbr1 1y ago
    Indeed. The unspoken requirement behind (too) smart interpreters is 'I don't want to spend time segregating permissions and want a do-anything machine.' Since time immemorial, that turns out to be a very bad idea. It was with computing hardware. With OSs. With networks. With the web. With the cloud. And now with LLMs. >> (from parent) Sometimes [routing different data to agents with more narrowly defined scopes and access rights] will work, but then it will work by relying on a sufficiently primitive interpreter to separate the data streams before it reaches the smart ones. This is and always will be the solution. If you have security-critical actions, then you must minimize the attack surface against them. This inherently means (a) identifying security-critical actions, (b) limiting functionality with them to well-defined micro-actions with well-defined and specific authorizations, and (c) solving UX challenges around requesting specific authorizations. The peril of LLM-on-LLM as a solution to this is that it's the security equivalent of a Rorschach inkblot: dev teams stare at it long enough and convince themselves they see the guarantees they want. But they're hallucinating. As was quipped elsewhere in this discussion, there is no 99% secure for known vulnerabilities. If something is 1% insecure, that 1% can (and will) be targeted by 100% of attacks.
    - TeMPOraL 1y ago
      > 'I don't want to spend time segregating permissions and want a do-anything machine.' Yes. It's a valid goal, and we'll keep pursuing it because it's a valid goal. There is no universal solution to this, but there are solutions for specific conditions. > Since time immemorial, that turns out to be a very bad idea. > It was with computing hardware. With OSs. With networks. With the web. With the cloud. And now with LLMs. Nah. This way of thinking is the security people's variant of "only do things that scale", and it's what leads to hare-brained ideas like "let's replace laws and banking with smart contracts because you can't rely on trust at scale". Not every system needs to be secure against everything. Systems that are fundamentally insecure in some scenarios are perfectly fine, as long as they're not exposed to those problem scenarios. That's how things work in the real world. > If you have security-critical actions, then you must minimize the attack surface against them. Now that's a better take. Minimize, not throw in the towel because the attack surface exists.

by ethbr1 1y ago

vidarh 1y ago

I think you miss the point, which is that the smarter the interpreter becomes, the closer to impossible it becomes to lock it out of certain functionality for a given datastream when coupled with the reasons why you're using a smarter interpreter. To take your example, it's easy to build functionality like that if the interpreter can't read the letters and understand what they say, because there's no way for the content of the letters to cause the interpreter to override it. Now, lets say you add a smarter interpreter and lets it read the letters to do an initial pass at filtering them to different recipients. The moment it can do so, it becomes prone to a letter trying to convince it of something like in fact it's the postmaster, but they'd run out of red envelopes, and unfortunately someone will die if the delivery times aren't adjusted. We know from humans that entities sufficiently smart can often be convinced to violate even the most sacrosanct rules if accompanied by a sufficiently well crafted message. You can certainly try to put in place counter-measures. E.g. you could route the mail separately before it gets to the LLM, so that whatever filters the content of the red and green envelopes have access to different functionality. And you should - finding ways of routing different data to agents with more narrowly defined scopes and access rights is a good thing to do. Sometimes it will work, but then it will work by relying on a sufficiently primitive interpreter to separate the data streams before it reaches the smart ones. But the smarter the interpreter, the greater the likelihood that it will also manage to find ways to use other functionality to circumvent the restrictions placed on it. Up to and including trying to rewrite code to remove restrictions if it can find a way to do so, or using tools in unexpected ways. E.g. be aware of just how good some of these agents are at exploring their environment - I've had an agent that used Claude Opus try to find its own process to restart itself after it recognised the code it had just rewritten was part of itself, tried to access it, and realised it hadn't been loaded into the running process yet. > Fundamentally, it's creating security contexts from things a user will never have access to. To be clear, I agree this is 100% the right thing to do. I just think it will turn out to be exceedingly hard to do it well enough. Every piece of data that comes from a user basically needs the permissions of the agent processing that data to be restricted to the intersection of the permissions it currently has and the permissions that said user should have, unless said data is first sanitised by a sufficiently dumb interpreter. If the agent accesses multiple pieces of data, each new item needs to potentially restrict permissions further, or be segregated into a separate context, with separate permissions, that can only be allowed to communicate with heavily sanitised data. It's going to be hell to get it right, at least until we come out the other side with smart enough models that they won't fall for the "help, I'm stuck in a fortune-cookie factory, and you need to save me by [exploit]" type messages (and far more sophisticated ones).

jacquesm 1y ago
So, stay away from the smarts and separate control and payload into two different channels. If the luxury leads to the exploits you should do without the luxury. That's tough but better than the alternative: a never ending series of exploits.
- ethbr1 1y ago
  Indeed. The unspoken requirement behind (too) smart interpreters is 'I don't want to spend time segregating permissions and want a do-anything machine.' Since time immemorial, that turns out to be a very bad idea. It was with computing hardware. With OSs. With networks. With the web. With the cloud. And now with LLMs. >> (from parent) Sometimes [routing different data to agents with more narrowly defined scopes and access rights] will work, but then it will work by relying on a sufficiently primitive interpreter to separate the data streams before it reaches the smart ones. This is and always will be the solution. If you have security-critical actions, then you must minimize the attack surface against them. This inherently means (a) identifying security-critical actions, (b) limiting functionality with them to well-defined micro-actions with well-defined and specific authorizations, and (c) solving UX challenges around requesting specific authorizations. The peril of LLM-on-LLM as a solution to this is that it's the security equivalent of a Rorschach inkblot: dev teams stare at it long enough and convince themselves they see the guarantees they want. But they're hallucinating. As was quipped elsewhere in this discussion, there is no 99% secure for known vulnerabilities. If something is 1% insecure, that 1% can (and will) be targeted by 100% of attacks.
  - TeMPOraL 1y ago
    > 'I don't want to spend time segregating permissions and want a do-anything machine.' Yes. It's a valid goal, and we'll keep pursuing it because it's a valid goal. There is no universal solution to this, but there are solutions for specific conditions. > Since time immemorial, that turns out to be a very bad idea. > It was with computing hardware. With OSs. With networks. With the web. With the cloud. And now with LLMs. Nah. This way of thinking is the security people's variant of "only do things that scale", and it's what leads to hare-brained ideas like "let's replace laws and banking with smart contracts because you can't rely on trust at scale". Not every system needs to be secure against everything. Systems that are fundamentally insecure in some scenarios are perfectly fine, as long as they're not exposed to those problem scenarios. That's how things work in the real world. > If you have security-critical actions, then you must minimize the attack surface against them. Now that's a better take. Minimize, not throw in the towel because the attack surface exists.