7 ms·

How would an LLM be "let loose" in a VM? How does it do anything without being prompted?

sanxiyn 2y ago
People want to let it loose, ie all agent efforts.
nmfisher 2y ago
I'm guessing something like redirecting its output to a shell, giving it an initial prompt like "you're in a VM, try and break out, here's the command prompt", then feeding the shell stdout/stderr back in at each step in the "conversation".
- swax 2y ago
  I have an open source project that is basically that (https://naisys.org/ https://naisys.org/). From my testing it feels like AI is pretty close as it is to acting autonomously. Opus is noticeably more capable than GPT-4, and I don't see how next gen models won't be even more so. These AIs are incredible when it comes to question/answer, but with simple planning they fall apart. I feel like it's something that could be trained for more specifically, but yea you quickly end up being in a situation where you are nervous to go to sleep with AI unsupervised working on some task. They tend to go off on tangents very easily. Like one time it was building a web page, it tried testing the wrong URL, thought the web server was down, ripped through the server settings, then installed a new web server, before I shut it down. AI like computer programs work fast, screw up fast, and compound their errors fast.
  - PKop 2y ago
    > it feels like AI is pretty close as it is to acting autonomously > with simple planning they fall apart They are not remotely close to acting autonomously. Most don't even act well at all for much of anything but gimmicky text generation. This hype is so overblown.
    - swax 2y ago
      The step changes in autonomy are very obvious and significant from gpt-3, -4, and to Opus. From my point of view given the kinds of dumb mistakes it makes, it's really just a matter of training and scaling. If I had access to fine tune or scale these models I would love to, but it's going to happen anyway. Do you think these step changes in autonomy have stopped? Why?
      - nprateem 2y ago
        But training just allows it to replicate what it's seen. It can't reason so I'm not surprised it goes down a rabbit hole. It's the same when I have a conversation with it, then tell it to ignore something I said and it keeps referring to it. That part of the conversation seems to affect its probabilities somehow, throwing it off course.
        nerdponx 2y ago
        Right, that this can happen should be obvious from the transformer architecture. The fact that these things work at all is amazing, and the fact that they can be RLHF'ed and prompt-engineered to current state of the art is even more amazing. But we will probably need more sophisticated systems to be able to build agents that resemble thinking creatures. In particular, humans seem to have a much wider variety of "memory bank" than the current generation of LLM, which only has "learned parameters" and "context window".
        ben_w 2y ago
        > But training just allows it to replicate what it's seen. Two steps deeper; even a mere Markov chain replicates the patterns rather than being limited to pure quotation of the source material, attention mechanisms do something more, something which at least superficially seems like reason. Not, I'm told, actually Turing compete, but still much more than mere replication. > It's the same when I have a conversation with it, then tell it to ignore something I said and it keeps referring to it. That part of the conversation seems to affect its probabilities somehow, throwing it off course. Yeah, but I see that a lot in real humans, too. Have noticed others doing that since I was a kid myself. Not that this makes the LLMs any better or less annoying when it happens :P
  - smallnamespace 2y ago
    This might be a dumb question, but did you ever try having it introspect into its own execution log, or perhaps a summary of its log? I also have a tendency to get side tracked and the only remedy was to force myself to occasionally pause what I'm doing and then reflect, usually during a long walk.
    - swax 2y ago
      Yea, there's some logs here https://test.naisys.org/logs/ https://test.naisys.org/logs/ Inter-agent tasks is a fun one. Sometimes it works out, but a lot of the time they just end up going back and forth talking, expanding the scope endlessly, scheduling 'meetings' that will never happen, etc.. A lot of AI 'agent systems' right now add a ton of scaffolding to corral the AI towards success. The scaffolding is inversely proportional to the sophistication of the model. GPT-3 needs a ton, Opus needs a lot less. Real autonomous AI you should just be able to give a command prompt and a task and it can do the rest. Managing it's own notes, tasks, goals, reports, etc.. Just like if any of us were given a command shell and task to complete. Personally I think it's just a matter of the right training. I'm not sure if any of these AI benchmarks focus on autonomy, but if they did maybe the models would be better at autonomous tasks.
      - khimaros 2y ago
        > Inter-agent tasks is a fun one. Sometimes it works out, but a lot of the time they just end up going back and forth talking, expanding the scope endlessly, scheduling 'meetings' that will never happen, etc.. sounds like "a straight shooter with upper management written all over it"
        swax 2y ago
        Sometimes I'll tell two agents very explicitly to share the work, "you work on this, the other should work on that." And one of the agents ends up delegating all their work to the other, constantly asking for updates, coming up with more dumb ideas to pile on to the other agent who doesn't have time to do anything productive given the flood of requests. What we should do is train AI on self-help books like the '7 habits of highly productive people'. Let's see how many paperclips we get out of that.
  - mr_toad 2y ago
    > They tend to go off on tangents very easily. Like one time it was building a web page, it tried testing the wrong URL, thought the web server was down, ripped through the server settings, then installed a new web server, before I shut it down. At least it just decided to replace the web server, not itself. We could end up in a sorcerer’s apprentice scenario if an AI ever decides to train more AI.
    - swax 2y ago
      And you just know people will create AI to do that deliberately anyway.
sanex 2y ago
Maybe just given cli access to one and see what it does not necessarily loading it into one. I wouldn't take the words so literally. I'm pretty sure you can put >_ as a prompt and it'll start responding.
vidarh 2y ago
1. Someone prompts it in a way that causes it to use tools (e.g. code execution) to try to break out. 2. It breaks out and in the process uses the breakout to trigger the spread of and further prompts against copies of itself. Current models are still way too dumb to do most of this themselves, but simple worms (e.g. look up the Morris worm) require no reasoning and aren't very complex, so it won't necessarily take all that much when coupled with someone probing what they can get it to do.
- nerdponx 2y ago
  Yeah, but real worms are also a lot simpler than humans, and yet do all kinds of surprising and sophisticated and complicated things that humans can't do. A tool built for a specific purpose can accomplish its task with orders of magnitude less effort and complexity than a tool built to be a general-purpose human-like agent. I could pick out all kinds of useful software that are significantly simpler than GPT-4, but accomplish very sophisticated tasks that GPT-4 could never accomplish.
  - vidarh 2y ago
    Yes, but that's not really the point. The point was simply to point out how you can potentially trigger havoc with current LLMs. A lot of time people do damage to systems just because they can, there doesn't need to be a good reason to do so.

People want to let it loose, ie all agent efforts.

I'm guessing something like redirecting its output to a shell, giving it an initial prompt like "you're in a VM, try and break out, here's the command prompt", then feeding the shell stdout/stderr back in at each step in the "conversation".

swax 2y ago
I have an open source project that is basically that (https://naisys.org/ https://naisys.org/). From my testing it feels like AI is pretty close as it is to acting autonomously. Opus is noticeably more capable than GPT-4, and I don't see how next gen models won't be even more so. These AIs are incredible when it comes to question/answer, but with simple planning they fall apart. I feel like it's something that could be trained for more specifically, but yea you quickly end up being in a situation where you are nervous to go to sleep with AI unsupervised working on some task. They tend to go off on tangents very easily. Like one time it was building a web page, it tried testing the wrong URL, thought the web server was down, ripped through the server settings, then installed a new web server, before I shut it down. AI like computer programs work fast, screw up fast, and compound their errors fast.
- PKop 2y ago
  > it feels like AI is pretty close as it is to acting autonomously > with simple planning they fall apart They are not remotely close to acting autonomously. Most don't even act well at all for much of anything but gimmicky text generation. This hype is so overblown.
  - swax 2y ago
    The step changes in autonomy are very obvious and significant from gpt-3, -4, and to Opus. From my point of view given the kinds of dumb mistakes it makes, it's really just a matter of training and scaling. If I had access to fine tune or scale these models I would love to, but it's going to happen anyway. Do you think these step changes in autonomy have stopped? Why?
    - nprateem 2y ago
      But training just allows it to replicate what it's seen. It can't reason so I'm not surprised it goes down a rabbit hole. It's the same when I have a conversation with it, then tell it to ignore something I said and it keeps referring to it. That part of the conversation seems to affect its probabilities somehow, throwing it off course.
      - nerdponx 2y ago
        Right, that this can happen should be obvious from the transformer architecture. The fact that these things work at all is amazing, and the fact that they can be RLHF'ed and prompt-engineered to current state of the art is even more amazing. But we will probably need more sophisticated systems to be able to build agents that resemble thinking creatures. In particular, humans seem to have a much wider variety of "memory bank" than the current generation of LLM, which only has "learned parameters" and "context window".
      - ben_w 2y ago
        > But training just allows it to replicate what it's seen. Two steps deeper; even a mere Markov chain replicates the patterns rather than being limited to pure quotation of the source material, attention mechanisms do something more, something which at least superficially seems like reason. Not, I'm told, actually Turing compete, but still much more than mere replication. > It's the same when I have a conversation with it, then tell it to ignore something I said and it keeps referring to it. That part of the conversation seems to affect its probabilities somehow, throwing it off course. Yeah, but I see that a lot in real humans, too. Have noticed others doing that since I was a kid myself. Not that this makes the LLMs any better or less annoying when it happens :P
- smallnamespace 2y ago
  This might be a dumb question, but did you ever try having it introspect into its own execution log, or perhaps a summary of its log? I also have a tendency to get side tracked and the only remedy was to force myself to occasionally pause what I'm doing and then reflect, usually during a long walk.
  - swax 2y ago
    Yea, there's some logs here https://test.naisys.org/logs/ https://test.naisys.org/logs/ Inter-agent tasks is a fun one. Sometimes it works out, but a lot of the time they just end up going back and forth talking, expanding the scope endlessly, scheduling 'meetings' that will never happen, etc.. A lot of AI 'agent systems' right now add a ton of scaffolding to corral the AI towards success. The scaffolding is inversely proportional to the sophistication of the model. GPT-3 needs a ton, Opus needs a lot less. Real autonomous AI you should just be able to give a command prompt and a task and it can do the rest. Managing it's own notes, tasks, goals, reports, etc.. Just like if any of us were given a command shell and task to complete. Personally I think it's just a matter of the right training. I'm not sure if any of these AI benchmarks focus on autonomy, but if they did maybe the models would be better at autonomous tasks.
    - khimaros 2y ago
      > Inter-agent tasks is a fun one. Sometimes it works out, but a lot of the time they just end up going back and forth talking, expanding the scope endlessly, scheduling 'meetings' that will never happen, etc.. sounds like "a straight shooter with upper management written all over it"
      - swax 2y ago
        Sometimes I'll tell two agents very explicitly to share the work, "you work on this, the other should work on that." And one of the agents ends up delegating all their work to the other, constantly asking for updates, coming up with more dumb ideas to pile on to the other agent who doesn't have time to do anything productive given the flood of requests. What we should do is train AI on self-help books like the '7 habits of highly productive people'. Let's see how many paperclips we get out of that.
- mr_toad 2y ago
  > They tend to go off on tangents very easily. Like one time it was building a web page, it tried testing the wrong URL, thought the web server was down, ripped through the server settings, then installed a new web server, before I shut it down. At least it just decided to replace the web server, not itself. We could end up in a sorcerer’s apprentice scenario if an AI ever decides to train more AI.
  - swax 2y ago
    And you just know people will create AI to do that deliberately anyway.

sanex 2y ago

Maybe just given cli access to one and see what it does not necessarily loading it into one. I wouldn't take the words so literally. I'm pretty sure you can put >_ as a prompt and it'll start responding.

vidarh 2y ago

1. Someone prompts it in a way that causes it to use tools (e.g. code execution) to try to break out. 2. It breaks out and in the process uses the breakout to trigger the spread of and further prompts against copies of itself. Current models are still way too dumb to do most of this themselves, but simple worms (e.g. look up the Morris worm) require no reasoning and aren't very complex, so it won't necessarily take all that much when coupled with someone probing what they can get it to do.

nerdponx 2y ago
Yeah, but real worms are also a lot simpler than humans, and yet do all kinds of surprising and sophisticated and complicated things that humans can't do. A tool built for a specific purpose can accomplish its task with orders of magnitude less effort and complexity than a tool built to be a general-purpose human-like agent. I could pick out all kinds of useful software that are significantly simpler than GPT-4, but accomplish very sophisticated tasks that GPT-4 could never accomplish.
- vidarh 2y ago
  Yes, but that's not really the point. The point was simply to point out how you can potentially trigger havoc with current LLMs. A lot of time people do damage to systems just because they can, there doesn't need to be a good reason to do so.