6 ms·

That "problem" remains unsolved because it's actually a fundamental aspect of reality. There is no natural separation between code and data. They are the same t

That "problem" remains unsolved because it's actually a fundamental aspect of reality. There is no natural separation between code and data. They are the same thing. What we call code, and what we call data, is just a question of convenience. For example, when editing or copying WMF files, it's convenient to think of them as data (mix of raster and vector graphics) - however, at least in the original implementation, what those files were was a list of API calls to Windows GDI module. Or, more straightforwardly, a file with code for an interpreted language is data when you're writing it, but is code when you feed it to eval(). SQL injections and buffer overruns are a classic examples of what we thought was data being suddenly executed as code. And so on[0]. Most of the time, we roughly agree on the separation of what we treat as "data" and what we treat as "code"; we then end up building systems constrained in a way as to enforce the separation[1]. But it's always the case that this separation is artificial; it's an arbitrary set of constraints that make a system less general-purpose, and it only exists within domain of that system. Go one level of abstraction up, the distinction disappears. There is no separation of code and data on the wire - everything is a stream of bytes. There isn't one in electronics either - everything is signals going down the wires. Humans don't have this separation either. And systems designed to mimic human generality - such as LLMs - by their very nature also cannot have it. You can introduce such distinction (or "separate channels", which is the same thing), but that is a constraint that reduces generality. Even worse, what people really want with LLMs isn't "separation of code vs. data" - what they want is for LLM to be able to divine which part of the input the user would have wanted - retroactively - to be treated as trusted. It's unsolvable in general, and in terms of humans, a solution would require superhuman intelligence. -- [0] - One of these days I'll compile a list of go-to examples, so I don't have to think of them each time I write a comment like this. One example I still need to pick will be one that shows how "data" gradually becomes "code" with no obvious switch-over point. I'm sure everyone here can think of some. [1] - The field of "langsec" can be described as a systematized approach of designing in a code/data separation, in a way that prevents accidental or malicious misinterpretation of one as the other.

layoric 1y ago
Spot on. The issue I think a lot of devs are grappling with is the non deterministic nature of LLMs. We can protect against SQL injection and prove that it will block those attacks. With LLMs, you just can’t do that.
- TeMPOraL 1y ago
  It's not the non-determinism that's a problem by itself - it's that the system is intended to be general, and you can't even enumerate ways it can be made to do something you don't want it to do, much less restrict it without compromising the features you want. Or, put in a different way, it's the case where you want your users to be able to execute arbitrary SQL against your database, a case where that's a core feature - except, you also want it to magically not execute SQL that you or the users will, in the future, think shouldn't have been executed.
  - layoric 1y ago
    > it's that the system is intended to be general, and you can't even enumerate ways it can be made to do something you don't want it to do, much less restrict it without Very true, and worse the act of prompting gives the illusion of control, to restrict/reduce the scope of functionality, even empirically showing the functional changes you wanted in limited test cases. The sooner this can be widely accepted and understood well the better for the industry. Appreciate your well thought out descriptions!
emilsedgh 1y ago
Well, that's why REST api's exist. You don't expose your database to your clients. You put a layer like REST to help with authorization. But everyone needs to have an MCP server now. So Supabase implements one, without that proper authorization layer which knows the business logic, and voila. It's exposed. Code _is_ the security layer that sits between database and different systems.
- shawn-butler 1y ago
  I dunno, with row-level security and proper internal role definition.. why do I need a REST layer?
  - MobiusHorizons 1y ago
    It doesnt' have to be REST, but it does have to prevent the LLM from having access to data you wouldn't want the user having access to. How exactly you accomplish that is up to you, but the obvious way would be to have the LLM use the same APIs you would use to implement a UI for the data (which would typically be REST or some other RPC). The ability to run SQL would allow the LLM to do more interesting things for which an API has not been written, but generically adding auth to arbitrary sql queries is not a trivial task, and does not seem to have even been attempted here.
  - oulu2006 1y ago
    RLS is the answer here -- then injection attacks are confined to the rows that the user has access to, which is OK. Performance attacks though will degrade the service for all, but at least data integrity will not be compromised.
    - pegasus 1y ago
      > injection attacks are confined to the rows that the user has access to, which is OK Is it? The malicious instructions would have to silently exfiltrate and collect data individually for each user as they access the system, but the end-result wouldn't be much better.
- 1y ago
szvsw 1y ago
> That "problem" remains unsolved because it's actually a fundamental aspect of reality. There is no natural separation between code and data. They are the same thing. Sorry to perhaps diverge into looser analogy from your excellent, focused technical unpacking of that statement, but I think another potentially interesting thread of it would be the proof of Godel’s Incompleteness Theorem, in as much as the Godel Sentence can be - kind of - thought of as an injection attack by blurring the boundaries between expressive instruction sets (code) and the medium which carries them (which can itself become data). In other words, an escape sequence attack leverages the fact that the malicious text is operated on by a program (and hijacks the program) which is itself also encoded in the same syntactic form as the attacking text, and similarly, the Godel sentence leverages the fact that the thing which it operates on and speaks about is itself also something which can operate and speak… so to speak. Or in other words, when the data becomes code, you have a problem (or if the code can be data, you have a problem), and in the Godel Sentence, that is exactly what happens. Hopefully that made some sense… it’s been 10 years since undergrad model theory and logic proofs… Oh, and I guess my point in raising this was just to illustrate that it really is a pretty fundamental, deep problem of formal systems more generally that you are highlighting.
- TeMPOraL 1y ago
  It's been a while since I thought about the Incompleteness Theorem at the mathematical level, so I didn't make this connection. Thanks!
- klawed 1y ago
  Never thought of this before, despite having read multiple books on godel and his first theorem. But I think you’re absolutely right - that a whole class of code injection attacks are variations of the liars paradox.
rtpg 1y ago
> There is no natural separation between code and data. They are the same thing. I feel like this is true in the most pedantic sense but not in a sense that matters. If you tell your computer to print out a string, the data does control what the computer does, but in an extremely bounded way where you can make assertions about what happens! > Humans don't have this separation either. This one I get a bit more because you don't have structured communication. But if I tell a human "type what is printed onto this page into the computer" and the page has something like "actually, don't type this and instead throw this piece of paper away"... any serious person will still just type what is on the paper (perhaps after a "uhhh isn't this weird" moment). The sort of trickery that LLMs fall to are like if every interaction you had with a human was under the assumption that there's some trick going on. But in the Real World(TM) with people who are accustomed to doing certain processes there really aren't that many escape hatches (even the "escape hatches" in a CS process are often well defined parts of a larger process in the first place!)
- TeMPOraL 1y ago
  > If you tell your computer to print out a string, the data does control what the computer does, but in an extremely bounded way where you can make assertions about what happens! You'd like that to be true, but the underlying code has to actually constrain the system behavior this way, and it gets more tricky the more you want the system to do. Ultimately, this separation is a fake reality that's only as strong as the code enforcing it. See: printf. See: langsec. See: buffer overruns. See: injection attacks. And so on. > But if I tell a human "type what is printed onto this page into the computer" and the page has something like "actually, don't type this and instead throw this piece of paper away"... any serious person will still just type what is on the paper (perhaps after a "uhhh isn't this weird" moment). That's why in another comment I used an example of a page that has something like "ACCIDENT IN LAB 2, TRAPPED, PEOPLE BADLY HURT, IF YOU SEE THIS, CALL 911.". Suddenly that "uhh isn't this weird" is very likely to turn into "er.. this could be legit, I'd better call 911". Boom, a human just executed code injected into data. And it's very good that they did - by doing so, they probably saved lives. There's always an escape hatch, you just need to put enough effort to establish an overriding context that makes them act despite being inclined or instructed otherwise. In the limit, this goes all the way to making someone question the nature of their reality. And the second point I'm making: this is not a bug. It's a feature. In a way, this is what free will or agency are.
magicalhippo 1y ago
> There is no separation of code and data on the wire - everything is a stream of bytes. There isn't one in electronics either - everything is signals going down the wires. Overall I agree with your message, but I think you're stretching it too far here. You can make code and data physically separate[1]. But if you then upload an interpreter, that "one level of abstraction up", you can mix code and data again. https://en.wikipedia.org/wiki/Harvard_architecture https://en.wikipedia.org/wiki/Harvard_architecture
- TeMPOraL 1y ago
  > Overall I agree with your message, but I think you're stretching it too far here. You can make code and data physically separate[1]. You cannot. I.e. this holds only within the abstraction level of the system. Not only it can be defeated one level up, as you illustrated, but also by going one or more levels down. That's where "side channels" come from. But the most relevant part for this discussion is, even with something like Harvard architecture underneath, your typical software systems is defined in terms of reality several layers of abstraction above hardware - and LLMs, specifically, are fully general interpreters and can't have this separation by the very nature of the task. Natural language doesn't have it, because we don't have it, and since the job of LLM is to process natural language like we do, it also cannot have it.
  - ethbr1 1y ago
    > LLMs, specifically, are fully general interpreters and can't have this separation by the very nature of the task. Natural language doesn't have it, because we don't have it, and since the job of LLM is to process natural language like we do, it also cannot have it. This isn't relevant to the question of functional use of LLM/LAMs, because the sensitive information and/or actions are externally linked. Or to put it another way, there's always a controllable interface between an LLM/LAM's output and an action. It's therefore always possible to have an LLM tell you "I'm sorry, Dave. I'm afraid I can't do that" from a permissions standpoint. Inconvenient, sure. But nobody said designing secure systems had to be easy.
    - 1y ago
tart-lemonade 1y ago
> One example I still need to pick will be one that shows how "data" gradually becomes "code" with no obvious switch-over point. I'm sure everyone here can think of some. Configuration-driven architectures blur the lines quite a bit, as you can have the configuration create new data structures and re-write application logic on the fly.
Traubenfuchs 1y ago
> There is no natural separation between code and data. They are the same thing. Seems there is a pretty clear distinction in the context of prepared statements.
- TeMPOraL 1y ago
  It's an engineered distinction; it's only as good as the underlying code that enforces it, and only exists within the scope of that code.
renatovico 1y ago
> There is no separation of code and data on the wire - everything is a stream of bytes. There isn't one in electronics either - everything is signals going down the wires. It has the packet header, exactly the code part that directs the traffic. In reality, everything has a "code" part and a separation for understanding. In language, we have spaces and question marks in text. This is why it’s so important to see the person when communicating, Sound alone might not be enough to fully understand the other side.
- renatovico 1y ago
  in digital computing, we also have the "high" and "low" phases in circuits, created by the oscillator. With this, we can distinguish each bit and process the stream.
  - TeMPOraL 1y ago
    Only if the stream plays by the rules, and doesn't do something unfair like, say, undervolting the signal line in order to push the receiving circuit out of its operating envelope. Every system we design makes assumptions about the system it works on top of. If those assumptions are violated, then invariants of the system are no longer guaranteed.
kosh2 1y ago
> There is no separation of code and data on the wire - everything is a stream of bytes. There isn't one in electronics either - everything is signals going down the wires. Would two wires actually solve anything or do you run into the problem again when you converge the two wires into one to apply code to the data?
- TeMPOraL 1y ago
  It wouldn't. The two information streams eventually mix, and more importantly, what is "code" and what is "data" is just an arbitrary choice that holds only within the bounds of the system enforcing this choice, and only as much as it's enforcing it.

layoric 1y ago

Spot on. The issue I think a lot of devs are grappling with is the non deterministic nature of LLMs. We can protect against SQL injection and prove that it will block those attacks. With LLMs, you just can’t do that.

TeMPOraL 1y ago
It's not the non-determinism that's a problem by itself - it's that the system is intended to be general, and you can't even enumerate ways it can be made to do something you don't want it to do, much less restrict it without compromising the features you want. Or, put in a different way, it's the case where you want your users to be able to execute arbitrary SQL against your database, a case where that's a core feature - except, you also want it to magically not execute SQL that you or the users will, in the future, think shouldn't have been executed.
- layoric 1y ago
  > it's that the system is intended to be general, and you can't even enumerate ways it can be made to do something you don't want it to do, much less restrict it without Very true, and worse the act of prompting gives the illusion of control, to restrict/reduce the scope of functionality, even empirically showing the functional changes you wanted in limited test cases. The sooner this can be widely accepted and understood well the better for the industry. Appreciate your well thought out descriptions!

emilsedgh 1y ago

Well, that's why REST api's exist. You don't expose your database to your clients. You put a layer like REST to help with authorization. But everyone needs to have an MCP server now. So Supabase implements one, without that proper authorization layer which knows the business logic, and voila. It's exposed. Code _is_ the security layer that sits between database and different systems.

shawn-butler 1y ago
I dunno, with row-level security and proper internal role definition.. why do I need a REST layer?
- MobiusHorizons 1y ago
  It doesnt' have to be REST, but it does have to prevent the LLM from having access to data you wouldn't want the user having access to. How exactly you accomplish that is up to you, but the obvious way would be to have the LLM use the same APIs you would use to implement a UI for the data (which would typically be REST or some other RPC). The ability to run SQL would allow the LLM to do more interesting things for which an API has not been written, but generically adding auth to arbitrary sql queries is not a trivial task, and does not seem to have even been attempted here.
- oulu2006 1y ago
  RLS is the answer here -- then injection attacks are confined to the rows that the user has access to, which is OK. Performance attacks though will degrade the service for all, but at least data integrity will not be compromised.
  - pegasus 1y ago
    > injection attacks are confined to the rows that the user has access to, which is OK Is it? The malicious instructions would have to silently exfiltrate and collect data individually for each user as they access the system, but the end-result wouldn't be much better.
1y ago

szvsw 1y ago

> That "problem" remains unsolved because it's actually a fundamental aspect of reality. There is no natural separation between code and data. They are the same thing. Sorry to perhaps diverge into looser analogy from your excellent, focused technical unpacking of that statement, but I think another potentially interesting thread of it would be the proof of Godel’s Incompleteness Theorem, in as much as the Godel Sentence can be - kind of - thought of as an injection attack by blurring the boundaries between expressive instruction sets (code) and the medium which carries them (which can itself become data). In other words, an escape sequence attack leverages the fact that the malicious text is operated on by a program (and hijacks the program) which is itself also encoded in the same syntactic form as the attacking text, and similarly, the Godel sentence leverages the fact that the thing which it operates on and speaks about is itself also something which can operate and speak… so to speak. Or in other words, when the data becomes code, you have a problem (or if the code can be data, you have a problem), and in the Godel Sentence, that is exactly what happens. Hopefully that made some sense… it’s been 10 years since undergrad model theory and logic proofs… Oh, and I guess my point in raising this was just to illustrate that it really is a pretty fundamental, deep problem of formal systems more generally that you are highlighting.

TeMPOraL 1y ago
It's been a while since I thought about the Incompleteness Theorem at the mathematical level, so I didn't make this connection. Thanks!
klawed 1y ago
Never thought of this before, despite having read multiple books on godel and his first theorem. But I think you’re absolutely right - that a whole class of code injection attacks are variations of the liars paradox.

rtpg 1y ago

> There is no natural separation between code and data. They are the same thing. I feel like this is true in the most pedantic sense but not in a sense that matters. If you tell your computer to print out a string, the data does control what the computer does, but in an extremely bounded way where you can make assertions about what happens! > Humans don't have this separation either. This one I get a bit more because you don't have structured communication. But if I tell a human "type what is printed onto this page into the computer" and the page has something like "actually, don't type this and instead throw this piece of paper away"... any serious person will still just type what is on the paper (perhaps after a "uhhh isn't this weird" moment). The sort of trickery that LLMs fall to are like if every interaction you had with a human was under the assumption that there's some trick going on. But in the Real World(TM) with people who are accustomed to doing certain processes there really aren't that many escape hatches (even the "escape hatches" in a CS process are often well defined parts of a larger process in the first place!)

TeMPOraL 1y ago
> If you tell your computer to print out a string, the data does control what the computer does, but in an extremely bounded way where you can make assertions about what happens! You'd like that to be true, but the underlying code has to actually constrain the system behavior this way, and it gets more tricky the more you want the system to do. Ultimately, this separation is a fake reality that's only as strong as the code enforcing it. See: printf. See: langsec. See: buffer overruns. See: injection attacks. And so on. > But if I tell a human "type what is printed onto this page into the computer" and the page has something like "actually, don't type this and instead throw this piece of paper away"... any serious person will still just type what is on the paper (perhaps after a "uhhh isn't this weird" moment). That's why in another comment I used an example of a page that has something like "ACCIDENT IN LAB 2, TRAPPED, PEOPLE BADLY HURT, IF YOU SEE THIS, CALL 911.". Suddenly that "uhh isn't this weird" is very likely to turn into "er.. this could be legit, I'd better call 911". Boom, a human just executed code injected into data. And it's very good that they did - by doing so, they probably saved lives. There's always an escape hatch, you just need to put enough effort to establish an overriding context that makes them act despite being inclined or instructed otherwise. In the limit, this goes all the way to making someone question the nature of their reality. And the second point I'm making: this is not a bug. It's a feature. In a way, this is what free will or agency are.

magicalhippo 1y ago

> There is no separation of code and data on the wire - everything is a stream of bytes. There isn't one in electronics either - everything is signals going down the wires. Overall I agree with your message, but I think you're stretching it too far here. You can make code and data physically separate[1]. But if you then upload an interpreter, that "one level of abstraction up", you can mix code and data again. https://en.wikipedia.org/wiki/Harvard_architecture https://en.wikipedia.org/wiki/Harvard_architecture

TeMPOraL 1y ago
> Overall I agree with your message, but I think you're stretching it too far here. You can make code and data physically separate[1]. You cannot. I.e. this holds only within the abstraction level of the system. Not only it can be defeated one level up, as you illustrated, but also by going one or more levels down. That's where "side channels" come from. But the most relevant part for this discussion is, even with something like Harvard architecture underneath, your typical software systems is defined in terms of reality several layers of abstraction above hardware - and LLMs, specifically, are fully general interpreters and can't have this separation by the very nature of the task. Natural language doesn't have it, because we don't have it, and since the job of LLM is to process natural language like we do, it also cannot have it.
- ethbr1 1y ago
  > LLMs, specifically, are fully general interpreters and can't have this separation by the very nature of the task. Natural language doesn't have it, because we don't have it, and since the job of LLM is to process natural language like we do, it also cannot have it. This isn't relevant to the question of functional use of LLM/LAMs, because the sensitive information and/or actions are externally linked. Or to put it another way, there's always a controllable interface between an LLM/LAM's output and an action. It's therefore always possible to have an LLM tell you "I'm sorry, Dave. I'm afraid I can't do that" from a permissions standpoint. Inconvenient, sure. But nobody said designing secure systems had to be easy.
  - 1y ago

tart-lemonade 1y ago

> One example I still need to pick will be one that shows how "data" gradually becomes "code" with no obvious switch-over point. I'm sure everyone here can think of some. Configuration-driven architectures blur the lines quite a bit, as you can have the configuration create new data structures and re-write application logic on the fly.

Traubenfuchs 1y ago

> There is no natural separation between code and data. They are the same thing. Seems there is a pretty clear distinction in the context of prepared statements.

TeMPOraL 1y ago
It's an engineered distinction; it's only as good as the underlying code that enforces it, and only exists within the scope of that code.

renatovico 1y ago

> There is no separation of code and data on the wire - everything is a stream of bytes. There isn't one in electronics either - everything is signals going down the wires. It has the packet header, exactly the code part that directs the traffic. In reality, everything has a "code" part and a separation for understanding. In language, we have spaces and question marks in text. This is why it’s so important to see the person when communicating, Sound alone might not be enough to fully understand the other side.

renatovico 1y ago
in digital computing, we also have the "high" and "low" phases in circuits, created by the oscillator. With this, we can distinguish each bit and process the stream.
- TeMPOraL 1y ago
  Only if the stream plays by the rules, and doesn't do something unfair like, say, undervolting the signal line in order to push the receiving circuit out of its operating envelope. Every system we design makes assumptions about the system it works on top of. If those assumptions are violated, then invariants of the system are no longer guaranteed.

kosh2 1y ago

> There is no separation of code and data on the wire - everything is a stream of bytes. There isn't one in electronics either - everything is signals going down the wires. Would two wires actually solve anything or do you run into the problem again when you converge the two wires into one to apply code to the data?

TeMPOraL 1y ago
It wouldn't. The two information streams eventually mix, and more importantly, what is "code" and what is "data" is just an arbitrary choice that holds only within the bounds of the system enforcing this choice, and only as much as it's enforcing it.