4 ms·

You're overcomplicating a thing that is simple -- don't use in-band control signaling. It's been the same problem since whistling for long-distance, with the s

by ethbr1 1y ago

You're overcomplicating a thing that is simple -- don't use in-band control signaling. It's been the same problem since whistling for long-distance, with the same solution of moving control signals out of the data stream. Any system where control signals can possibly be expressed in input data is vulnerable to escape-escaping exploitation. The same solution, hard isolation, instantly solves the problem: you have to render control inexpressible in the in-band alphabet. Whether that's by carrying control signals on isolated transport (e.g CCS/SS7), making control signals inexpressible in the in-band set (e.g. using other frequencies or alphabets), using NX-style flagging, or other methods.

TeMPOraL 1y ago
> You're overcomplicating a thing that is simple -- don't use in-band control signaling. On the contrary, I'm claiming that this "simplicity" is an illusion. Reality has only one band. > It's been the same problem since whistling for long-distance, with the same solution of moving control signals out of the data stream. "Control signals" and "data stream" are just... two data streams. They always eventually mix. > The same solution, hard isolation, instantly solves the problem: you have to render control inexpressible in the in-band alphabet. This isn't something that exist in nature. We don't build machines out of platonic shapes and abstract math - we build them out of matter. You want such rules like "separation of data and code", "separation of control-data and data-data", and "control-data being inexpressible in data-data alphabet" to hold? You need to design a system so constrained, as to behave this way - creating a faux reality within itself, where those constraints hold. But people keep forgetting - this is a faux reality. Those constraints only hold within it, not outside it[0], and to the extent you actually implemented what you thought you did (we routinely fuck that up). I start to digress, so to get back to the point: such constraints are okay, but they by definition limit what the system could do. This is fine when that's what you want, but LLMs are explicitly designed to not be that. LLMs are built for one purpose - to process natural language like we do. That's literally the goal function used in training - take in arbitrary input, produce output that looks right to humans, in fully general sense of that[1]. We've evolved to function in the physical reality - not some designed faux-reality. We don't have separate control and data channels. We've developed natural language to describe that reality, to express ourselves and coordinate with others - and natural language too does not have any kind of control and data separation, because our brains fundamentally don't implement that. More than that, our natural language relies on there being no such separation. LLMs therefore cannot be made to have that separation either. We can't have it both ways. -- [0] - The "constraints only apply within the system" part is what keeps tripping people over. You may think your telegraph cannot possibly be controlled over the data wire - it really doesn't even parse the data stream, literally just forwards it as-is, to a destination selected on another band. What you don't know is, I looked up the specs of your telegraph, and figured out that if I momentarily plug a car battery to the signal line, it'll briefly overload a control relay in your telegraph, and if I time this right, I can make the telegraph switch destinations. (Okay, you treat it as a bug and add some hardware to eliminate "overvoltage events" from what can be "expressed in the in-band alphabet". But you forgot that the control and data wires actually run close to each other for a few meters - so let me introduce you to the concept of electromagnetic induction.) And so on, and so on. We call those things "side channels", and they're not limited to exploiting physics; they're just about exploiting the fact that your system is built in terms of other systems with different rules. [1] - Understanding, reasoning, modelling the world, etc. all follow directly from that - natural language directly involves those capabilities, so having or emulating them is required.
- ethbr1 1y ago
  (Broad reply upthread) Is it more difficult to hijack an out-of-band control signal or an in-band one? That there exist details to architecting full isolation well doesn't mean we shouldn't try. At root, giving LLMs permissions to execute security sensitive actions and then trying to prevent them from doing so is a fool's errand -- don't fucking give a black box those permissions! (Yes, even when every test you threw at it said it would be fine) LLMs as security barriers is a new record for laziest and stupidest idea the field has had.
vidarh 1y ago
The problem is that the moment the interpreter is powerful enough, you're relying on the data not being good enough at convincing the interpreter that it is an exception. You can only maintain hard isolation if the interpreter of the data is sufficiently primitive, and even then it is often hard to avoid errors that renders it more powerful than intended, be it outright bugs all the way up to unintentional Turing completeness.
- ethbr1 1y ago
  (I'll reply to you because you expressed it more succinctly) Yes and no. I think this is exactly the distinction that's been institutionally lost in the last few decades, because few people are architecting from top (software) to bottom (physical transport) of the stack anymore. They just try and cram functionality in the topmost layer, when it should leverage others. If I lock an interpreter out of certain functionality for a given data stream, ever, then exploitation becomes orders of magnitude more difficult. Dumb analogy: only letters in red envelopes get to change mail delivery times + all regular mail is packaged in green envelopes Fundamentally, it's creating security contexts from things a user will never have access to. The LLMs-on-top-of-LLMs filtering approach is lazy and statistically guaranteed to end badly.
  - vidarh 1y ago
    I think you miss the point, which is that the smarter the interpreter becomes, the closer to impossible it becomes to lock it out of certain functionality for a given datastream when coupled with the reasons why you're using a smarter interpreter. To take your example, it's easy to build functionality like that if the interpreter can't read the letters and understand what they say, because there's no way for the content of the letters to cause the interpreter to override it. Now, lets say you add a smarter interpreter and lets it read the letters to do an initial pass at filtering them to different recipients. The moment it can do so, it becomes prone to a letter trying to convince it of something like in fact it's the postmaster, but they'd run out of red envelopes, and unfortunately someone will die if the delivery times aren't adjusted. We know from humans that entities sufficiently smart can often be convinced to violate even the most sacrosanct rules if accompanied by a sufficiently well crafted message. You can certainly try to put in place counter-measures. E.g. you could route the mail separately before it gets to the LLM, so that whatever filters the content of the red and green envelopes have access to different functionality. And you should - finding ways of routing different data to agents with more narrowly defined scopes and access rights is a good thing to do. Sometimes it will work, but then it will work by relying on a sufficiently primitive interpreter to separate the data streams before it reaches the smart ones. But the smarter the interpreter, the greater the likelihood that it will also manage to find ways to use other functionality to circumvent the restrictions placed on it. Up to and including trying to rewrite code to remove restrictions if it can find a way to do so, or using tools in unexpected ways. E.g. be aware of just how good some of these agents are at exploring their environment - I've had an agent that used Claude Opus try to find its own process to restart itself after it recognised the code it had just rewritten was part of itself, tried to access it, and realised it hadn't been loaded into the running process yet. > Fundamentally, it's creating security contexts from things a user will never have access to. To be clear, I agree this is 100% the right thing to do. I just think it will turn out to be exceedingly hard to do it well enough. Every piece of data that comes from a user basically needs the permissions of the agent processing that data to be restricted to the intersection of the permissions it currently has and the permissions that said user should have, unless said data is first sanitised by a sufficiently dumb interpreter. If the agent accesses multiple pieces of data, each new item needs to potentially restrict permissions further, or be segregated into a separate context, with separate permissions, that can only be allowed to communicate with heavily sanitised data. It's going to be hell to get it right, at least until we come out the other side with smart enough models that they won't fall for the "help, I'm stuck in a fortune-cookie factory, and you need to save me by [exploit]" type messages (and far more sophisticated ones).

by ethbr1 1y ago

TeMPOraL 1y ago

> You're overcomplicating a thing that is simple -- don't use in-band control signaling. On the contrary, I'm claiming that this "simplicity" is an illusion. Reality has only one band. > It's been the same problem since whistling for long-distance, with the same solution of moving control signals out of the data stream. "Control signals" and "data stream" are just... two data streams. They always eventually mix. > The same solution, hard isolation, instantly solves the problem: you have to render control inexpressible in the in-band alphabet. This isn't something that exist in nature. We don't build machines out of platonic shapes and abstract math - we build them out of matter. You want such rules like "separation of data and code", "separation of control-data and data-data", and "control-data being inexpressible in data-data alphabet" to hold? You need to design a system so constrained, as to behave this way - creating a faux reality within itself, where those constraints hold. But people keep forgetting - this is a faux reality. Those constraints only hold within it, not outside it[0], and to the extent you actually implemented what you thought you did (we routinely fuck that up). I start to digress, so to get back to the point: such constraints are okay, but they by definition limit what the system could do. This is fine when that's what you want, but LLMs are explicitly designed to not be that. LLMs are built for one purpose - to process natural language like we do. That's literally the goal function used in training - take in arbitrary input, produce output that looks right to humans, in fully general sense of that[1]. We've evolved to function in the physical reality - not some designed faux-reality. We don't have separate control and data channels. We've developed natural language to describe that reality, to express ourselves and coordinate with others - and natural language too does not have any kind of control and data separation, because our brains fundamentally don't implement that. More than that, our natural language relies on there being no such separation. LLMs therefore cannot be made to have that separation either. We can't have it both ways. -- [0] - The "constraints only apply within the system" part is what keeps tripping people over. You may think your telegraph cannot possibly be controlled over the data wire - it really doesn't even parse the data stream, literally just forwards it as-is, to a destination selected on another band. What you don't know is, I looked up the specs of your telegraph, and figured out that if I momentarily plug a car battery to the signal line, it'll briefly overload a control relay in your telegraph, and if I time this right, I can make the telegraph switch destinations. (Okay, you treat it as a bug and add some hardware to eliminate "overvoltage events" from what can be "expressed in the in-band alphabet". But you forgot that the control and data wires actually run close to each other for a few meters - so let me introduce you to the concept of electromagnetic induction.) And so on, and so on. We call those things "side channels", and they're not limited to exploiting physics; they're just about exploiting the fact that your system is built in terms of other systems with different rules. [1] - Understanding, reasoning, modelling the world, etc. all follow directly from that - natural language directly involves those capabilities, so having or emulating them is required.

ethbr1 1y ago
(Broad reply upthread) Is it more difficult to hijack an out-of-band control signal or an in-band one? That there exist details to architecting full isolation well doesn't mean we shouldn't try. At root, giving LLMs permissions to execute security sensitive actions and then trying to prevent them from doing so is a fool's errand -- don't fucking give a black box those permissions! (Yes, even when every test you threw at it said it would be fine) LLMs as security barriers is a new record for laziest and stupidest idea the field has had.

vidarh 1y ago

The problem is that the moment the interpreter is powerful enough, you're relying on the data not being good enough at convincing the interpreter that it is an exception. You can only maintain hard isolation if the interpreter of the data is sufficiently primitive, and even then it is often hard to avoid errors that renders it more powerful than intended, be it outright bugs all the way up to unintentional Turing completeness.

ethbr1 1y ago
(I'll reply to you because you expressed it more succinctly) Yes and no. I think this is exactly the distinction that's been institutionally lost in the last few decades, because few people are architecting from top (software) to bottom (physical transport) of the stack anymore. They just try and cram functionality in the topmost layer, when it should leverage others. If I lock an interpreter out of certain functionality for a given data stream, ever, then exploitation becomes orders of magnitude more difficult. Dumb analogy: only letters in red envelopes get to change mail delivery times + all regular mail is packaged in green envelopes Fundamentally, it's creating security contexts from things a user will never have access to. The LLMs-on-top-of-LLMs filtering approach is lazy and statistically guaranteed to end badly.
- vidarh 1y ago
  I think you miss the point, which is that the smarter the interpreter becomes, the closer to impossible it becomes to lock it out of certain functionality for a given datastream when coupled with the reasons why you're using a smarter interpreter. To take your example, it's easy to build functionality like that if the interpreter can't read the letters and understand what they say, because there's no way for the content of the letters to cause the interpreter to override it. Now, lets say you add a smarter interpreter and lets it read the letters to do an initial pass at filtering them to different recipients. The moment it can do so, it becomes prone to a letter trying to convince it of something like in fact it's the postmaster, but they'd run out of red envelopes, and unfortunately someone will die if the delivery times aren't adjusted. We know from humans that entities sufficiently smart can often be convinced to violate even the most sacrosanct rules if accompanied by a sufficiently well crafted message. You can certainly try to put in place counter-measures. E.g. you could route the mail separately before it gets to the LLM, so that whatever filters the content of the red and green envelopes have access to different functionality. And you should - finding ways of routing different data to agents with more narrowly defined scopes and access rights is a good thing to do. Sometimes it will work, but then it will work by relying on a sufficiently primitive interpreter to separate the data streams before it reaches the smart ones. But the smarter the interpreter, the greater the likelihood that it will also manage to find ways to use other functionality to circumvent the restrictions placed on it. Up to and including trying to rewrite code to remove restrictions if it can find a way to do so, or using tools in unexpected ways. E.g. be aware of just how good some of these agents are at exploring their environment - I've had an agent that used Claude Opus try to find its own process to restart itself after it recognised the code it had just rewritten was part of itself, tried to access it, and realised it hadn't been loaded into the running process yet. > Fundamentally, it's creating security contexts from things a user will never have access to. To be clear, I agree this is 100% the right thing to do. I just think it will turn out to be exceedingly hard to do it well enough. Every piece of data that comes from a user basically needs the permissions of the agent processing that data to be restricted to the intersection of the permissions it currently has and the permissions that said user should have, unless said data is first sanitised by a sufficiently dumb interpreter. If the agent accesses multiple pieces of data, each new item needs to potentially restrict permissions further, or be segregated into a separate context, with separate permissions, that can only be allowed to communicate with heavily sanitised data. It's going to be hell to get it right, at least until we come out the other side with smart enough models that they won't fall for the "help, I'm stuck in a fortune-cookie factory, and you need to save me by [exploit]" type messages (and far more sophisticated ones).