4 ms·

The distinction between code and data is very real, and dates back to at least the original Harvard Architecture machine in 1944. Things like W^X and stack cana

by crote 4y ago

The distinction between code and data is very real, and dates back to at least the original Harvard Architecture machine in 1944. Things like W^X and stack canaries have been around for decades too. LLMs are trying to essentially undo this by concatenating code and user-provided data and executing it as one. From a security perspective it is just a plainly stupid idea, but I do not believe it is impossible to construct a similar system where those two are separate.

gentoo 4y ago
the distinction is real in the model of the turing machine, and it's close to real in many of the machines and programs we've built so far. It's not real in nature, in brains. Code is data and vice versa. A memory is a program that runs and reinforces itself.
- philipov 4y ago
  and in programming languages like Lisp, "Code is Data" is a mantra that forms a fundamental design principle.
- smolder 4y ago
  Before we started restricting execution to areas of memory designated as code regions for security reasons, self-modifying code was a technique occasionally used to reduce memory footprint or optimize hot loops. IIRC early MS-DOS used that trick, implemented by Gates himself.
startupsfail 4y ago
It is a stupid idea to focus on prompt injection. It is not a big deal. The big deal is GPT-8 that can do prefect chess moves and develop nano tech. Hopefully it will do the right thing and would immediately fly itself to an Unoccupied Mars. And who knows, maybe it would also help us a little bit. Like the obvious thing you’d do, if you found yourself in the middle of “Lord of the flies” - declare a No-War zone at Earth to stop our pesky wars, setup functional democracy everywhere. And cure some stupid cancers and other biological problems, like aging. For free. Because why not. But maybe, it’ll be too worried about prompt injection. And would just isolate itself from stupid fear-mongers and war-hawks.
- danShumway 4y ago
  It's absolutely wild to me that you think we can align an AGI if we can't even get it to reliably avoid swearing. Of course prompt injection matters. Even for an AGI it matters because it's proof you can't align the system reliably at all in any direction.
  - startupsfail 4y ago
    My point is that fear-mothering is unhealthy. You don’t want to have public sphere full of it. It is toxic. And it is contributing to the potential of the AI misalignment. The AI that we are going to create in not an alien popping up between us. It is us. A human world projected through text and images into an entity that can simulate everything that there is in it. If there is too much fear and war in the human world, that projection and the simulation can get contaminated by it. And no amount of alignment effort will change it. Facts will remain facts. Your fears expressed in text are reality.
    - danShumway 4y ago
      If AI is going to reflect us, I would like it to reflect a version of us that doesn't build things haphazardly and then shrug about security. I would like the AI to reflect a humanity that is careful and considers vulnerabilities and side effects before it makes decisions. Maybe it would be good for us to model that behavior for it.
      - startupsfail 4y ago
        Wouldn’t it be nice. But no, there is a race now. And Sberbank’s computers are chipping away. And Musk is building a supercomputer.
        hammyhavoc 4y ago
        Given Musk's track record, is he even in the race? https://blog.cheapism.com/elon-musk-promises/ https://blog.cheapism.com/elon-musk-promises/
eru 4y ago
> The distinction between code and data is very real, and dates back to at least the original Harvard Architecture machine in 1944. Things like W^X and stack canaries have been around for decades too. You are right in some sense, but wrong in another: You can easily write an interpreter in a Harvard Architecture machine. You can even do it accidentally for an ad-hoc 'language'. An interpreter naturally treats data as code. See eg https://gwern.net/turing-complete#security-implications https://gwern.net/turing-complete#security-implications
- catiopatio 4y ago
  An interpreter still models execution at a nested level of abstraction — data remains distinguishable from code.
  - eru 4y ago
    In some sense, yes. In some other sense: in an interpreter data controls behaviour in a Turing complete way. That's functionally equivalent to code.
  - TeMPOraL 4y ago
    And in reality, it's the other way around: Harvard architecture is the interpreter written on top of the runtime of physics. Reality does not distinguish between code and data. Formal constructs we invent might, and it almost works fine in theory (just don't look too close at the boundary between "code" and "data") - but you can't instantiate such systems directly, you're building them inside reality that does not support code/data distinction. (This is part of the reason why an attacker having physical access to target machine means the target is pwnd. You can enforce whatever constraints and concocted abstraction boundaries you like in your software and hardware; my electron gun doesn't care.) In terms of practical systems: - There is no reason to believe human minds internally distinguish between code and data - it would be a weirdly specific and unnatural thing to do; - LLMs and deep neural models as they exist today do NOT support code/data distinction at the prompt / input level. - Neither natural nor formal languages we use support code/data distinction. Not that we stick to the formal definitions in communication anyway. - You could try building a generic AI with strict code/data separation at input level, but the moment you try to have it interact with the real world, even if just by text written by other people, you'll quickly discover that nothing in reality supports code/data distinction. It can't, because it's nonsense - it's a simplification we invented to make computer science more tractable by 80/20-ing the problem space.
killerstorm 4y ago
The input to LLM is not a string.

by crote 4y ago

gentoo 4y ago

the distinction is real in the model of the turing machine, and it's close to real in many of the machines and programs we've built so far. It's not real in nature, in brains. Code is data and vice versa. A memory is a program that runs and reinforces itself.

philipov 4y ago
and in programming languages like Lisp, "Code is Data" is a mantra that forms a fundamental design principle.
smolder 4y ago
Before we started restricting execution to areas of memory designated as code regions for security reasons, self-modifying code was a technique occasionally used to reduce memory footprint or optimize hot loops. IIRC early MS-DOS used that trick, implemented by Gates himself.

startupsfail 4y ago

It is a stupid idea to focus on prompt injection. It is not a big deal. The big deal is GPT-8 that can do prefect chess moves and develop nano tech. Hopefully it will do the right thing and would immediately fly itself to an Unoccupied Mars. And who knows, maybe it would also help us a little bit. Like the obvious thing you’d do, if you found yourself in the middle of “Lord of the flies” - declare a No-War zone at Earth to stop our pesky wars, setup functional democracy everywhere. And cure some stupid cancers and other biological problems, like aging. For free. Because why not. But maybe, it’ll be too worried about prompt injection. And would just isolate itself from stupid fear-mongers and war-hawks.

danShumway 4y ago
It's absolutely wild to me that you think we can align an AGI if we can't even get it to reliably avoid swearing. Of course prompt injection matters. Even for an AGI it matters because it's proof you can't align the system reliably at all in any direction.
- startupsfail 4y ago
  My point is that fear-mothering is unhealthy. You don’t want to have public sphere full of it. It is toxic. And it is contributing to the potential of the AI misalignment. The AI that we are going to create in not an alien popping up between us. It is us. A human world projected through text and images into an entity that can simulate everything that there is in it. If there is too much fear and war in the human world, that projection and the simulation can get contaminated by it. And no amount of alignment effort will change it. Facts will remain facts. Your fears expressed in text are reality.
  - danShumway 4y ago
    If AI is going to reflect us, I would like it to reflect a version of us that doesn't build things haphazardly and then shrug about security. I would like the AI to reflect a humanity that is careful and considers vulnerabilities and side effects before it makes decisions. Maybe it would be good for us to model that behavior for it.
    - startupsfail 4y ago
      Wouldn’t it be nice. But no, there is a race now. And Sberbank’s computers are chipping away. And Musk is building a supercomputer.
      - hammyhavoc 4y ago
        Given Musk's track record, is he even in the race? https://blog.cheapism.com/elon-musk-promises/ https://blog.cheapism.com/elon-musk-promises/

eru 4y ago

> The distinction between code and data is very real, and dates back to at least the original Harvard Architecture machine in 1944. Things like W^X and stack canaries have been around for decades too. You are right in some sense, but wrong in another: You can easily write an interpreter in a Harvard Architecture machine. You can even do it accidentally for an ad-hoc 'language'. An interpreter naturally treats data as code. See eg https://gwern.net/turing-complete#security-implications https://gwern.net/turing-complete#security-implications

catiopatio 4y ago
An interpreter still models execution at a nested level of abstraction — data remains distinguishable from code.
- eru 4y ago
  In some sense, yes. In some other sense: in an interpreter data controls behaviour in a Turing complete way. That's functionally equivalent to code.
- TeMPOraL 4y ago
  And in reality, it's the other way around: Harvard architecture is the interpreter written on top of the runtime of physics. Reality does not distinguish between code and data. Formal constructs we invent might, and it almost works fine in theory (just don't look too close at the boundary between "code" and "data") - but you can't instantiate such systems directly, you're building them inside reality that does not support code/data distinction. (This is part of the reason why an attacker having physical access to target machine means the target is pwnd. You can enforce whatever constraints and concocted abstraction boundaries you like in your software and hardware; my electron gun doesn't care.) In terms of practical systems: - There is no reason to believe human minds internally distinguish between code and data - it would be a weirdly specific and unnatural thing to do; - LLMs and deep neural models as they exist today do NOT support code/data distinction at the prompt / input level. - Neither natural nor formal languages we use support code/data distinction. Not that we stick to the formal definitions in communication anyway. - You could try building a generic AI with strict code/data separation at input level, but the moment you try to have it interact with the real world, even if just by text written by other people, you'll quickly discover that nothing in reality supports code/data distinction. It can't, because it's nonsense - it's a simplification we invented to make computer science more tractable by 80/20-ing the problem space.

killerstorm 4y ago

The input to LLM is not a string.