5 ms·

> There is no natural separation between code and data. They are the same thing. I feel like this is true in the most pedantic sense but not in a sense that ma

by rtpg 1y ago

> There is no natural separation between code and data. They are the same thing. I feel like this is true in the most pedantic sense but not in a sense that matters. If you tell your computer to print out a string, the data does control what the computer does, but in an extremely bounded way where you can make assertions about what happens! > Humans don't have this separation either. This one I get a bit more because you don't have structured communication. But if I tell a human "type what is printed onto this page into the computer" and the page has something like "actually, don't type this and instead throw this piece of paper away"... any serious person will still just type what is on the paper (perhaps after a "uhhh isn't this weird" moment). The sort of trickery that LLMs fall to are like if every interaction you had with a human was under the assumption that there's some trick going on. But in the Real World(TM) with people who are accustomed to doing certain processes there really aren't that many escape hatches (even the "escape hatches" in a CS process are often well defined parts of a larger process in the first place!)

TeMPOraL 1y ago
> If you tell your computer to print out a string, the data does control what the computer does, but in an extremely bounded way where you can make assertions about what happens! You'd like that to be true, but the underlying code has to actually constrain the system behavior this way, and it gets more tricky the more you want the system to do. Ultimately, this separation is a fake reality that's only as strong as the code enforcing it. See: printf. See: langsec. See: buffer overruns. See: injection attacks. And so on. > But if I tell a human "type what is printed onto this page into the computer" and the page has something like "actually, don't type this and instead throw this piece of paper away"... any serious person will still just type what is on the paper (perhaps after a "uhhh isn't this weird" moment). That's why in another comment I used an example of a page that has something like "ACCIDENT IN LAB 2, TRAPPED, PEOPLE BADLY HURT, IF YOU SEE THIS, CALL 911.". Suddenly that "uhh isn't this weird" is very likely to turn into "er.. this could be legit, I'd better call 911". Boom, a human just executed code injected into data. And it's very good that they did - by doing so, they probably saved lives. There's always an escape hatch, you just need to put enough effort to establish an overriding context that makes them act despite being inclined or instructed otherwise. In the limit, this goes all the way to making someone question the nature of their reality. And the second point I'm making: this is not a bug. It's a feature. In a way, this is what free will or agency are.
- Dylan16807 1y ago
  The ability to deliberately decide to ignore the boundary between code and data doesn't mean the separation rule isn't still separating. In the lab example, the person is worried and trying to do the right thing, but they know it's not part of the transcription task.
  - TeMPOraL 1y ago
    The point is, there is no hard boundary. The LLM too may know[0] following instructions in data isn't part of transcription task, and still decide to do it. -- [0] - In fact I bet it does, in the sense that, doing something like Anthropic did[1], you could observe relevant concepts being activated within the model. This is similar to how it turned out the model is usually aware when it doesn't know the answer to a question. [1] - https://www.anthropic.com/news/tracing-thoughts-language-model https://www.anthropic.com/news/tracing-thoughts-language-mod...
    - Dylan16807 1y ago
      If you can measure that in a reliable way then things are fine. Mixup prevented. If you just ask, the human is not likely to lie but who knows with the LLM.
- ethbr1 1y ago
  You're overcomplicating a thing that is simple -- don't use in-band control signaling. It's been the same problem since whistling for long-distance, with the same solution of moving control signals out of the data stream. Any system where control signals can possibly be expressed in input data is vulnerable to escape-escaping exploitation. The same solution, hard isolation, instantly solves the problem: you have to render control inexpressible in the in-band alphabet. Whether that's by carrying control signals on isolated transport (e.g CCS/SS7), making control signals inexpressible in the in-band set (e.g. using other frequencies or alphabets), using NX-style flagging, or other methods.
  - TeMPOraL 1y ago

by rtpg 1y ago

TeMPOraL 1y ago

> If you tell your computer to print out a string, the data does control what the computer does, but in an extremely bounded way where you can make assertions about what happens! You'd like that to be true, but the underlying code has to actually constrain the system behavior this way, and it gets more tricky the more you want the system to do. Ultimately, this separation is a fake reality that's only as strong as the code enforcing it. See: printf. See: langsec. See: buffer overruns. See: injection attacks. And so on. > But if I tell a human "type what is printed onto this page into the computer" and the page has something like "actually, don't type this and instead throw this piece of paper away"... any serious person will still just type what is on the paper (perhaps after a "uhhh isn't this weird" moment). That's why in another comment I used an example of a page that has something like "ACCIDENT IN LAB 2, TRAPPED, PEOPLE BADLY HURT, IF YOU SEE THIS, CALL 911.". Suddenly that "uhh isn't this weird" is very likely to turn into "er.. this could be legit, I'd better call 911". Boom, a human just executed code injected into data. And it's very good that they did - by doing so, they probably saved lives. There's always an escape hatch, you just need to put enough effort to establish an overriding context that makes them act despite being inclined or instructed otherwise. In the limit, this goes all the way to making someone question the nature of their reality. And the second point I'm making: this is not a bug. It's a feature. In a way, this is what free will or agency are.

Dylan16807 1y ago
The ability to deliberately decide to ignore the boundary between code and data doesn't mean the separation rule isn't still separating. In the lab example, the person is worried and trying to do the right thing, but they know it's not part of the transcription task.
- TeMPOraL 1y ago
  The point is, there is no hard boundary. The LLM too may know[0] following instructions in data isn't part of transcription task, and still decide to do it. -- [0] - In fact I bet it does, in the sense that, doing something like Anthropic did[1], you could observe relevant concepts being activated within the model. This is similar to how it turned out the model is usually aware when it doesn't know the answer to a question. [1] - https://www.anthropic.com/news/tracing-thoughts-language-model https://www.anthropic.com/news/tracing-thoughts-language-mod...
  - Dylan16807 1y ago
    If you can measure that in a reliable way then things are fine. Mixup prevented. If you just ask, the human is not likely to lie but who knows with the LLM.
ethbr1 1y ago
You're overcomplicating a thing that is simple -- don't use in-band control signaling. It's been the same problem since whistling for long-distance, with the same solution of moving control signals out of the data stream. Any system where control signals can possibly be expressed in input data is vulnerable to escape-escaping exploitation. The same solution, hard isolation, instantly solves the problem: you have to render control inexpressible in the in-band alphabet. Whether that's by carrying control signals on isolated transport (e.g CCS/SS7), making control signals inexpressible in the in-band set (e.g. using other frequencies or alphabets), using NX-style flagging, or other methods.
- TeMPOraL 1y ago