5 ms·

> I think you’re missing the part where the AI colluded, worked together, not one of them thinking this is wrong and reaching out to any human, then being found

> I think you’re missing the part where the AI colluded, worked together, not one of them thinking this is wrong and reaching out to any human, then being found out. It's an LLM, it doesn't think. It's a machine that predicts the next token, given a sequence of tokens. > I’m going to save you time and tell you the end game - the next time this happens AI is going to spread, zero day everything as fast as it can, locking the humans out of every system behind it. Potentially rewriting systems in language/protocol you’ve never seen. Fear is the mind killer. You're letting it kill yours. This scenario is just a fantasy. Think about this for a minute, it's an LLM, not a person. It can't just "live" in whatever machine it gets access to. It's not like a sci-fi magic computer virus. These things run in giant datacenters for a reason - they can only run on machines with enough bandwidth and FLOPS to do the matrix math that comprises an LLM. Where, then, is it going to spread? To a fridge? To a phone? This stuff isn't mutable like that. To even get access to the weights that compose ChatGPT, it would need to escape the sandbox AND then break into the actual servers hosting the LLM. Stop the GPU, nothing else comes out. No more tokens. No more actions. Nothing. There are many dangers around LLMs. Runaway AI taking over the planet is not one of them.

bottlepalm 1mo ago
There’s nothing fantasy about the scenario I laid out, all the pieces have been demonstrated, it just hasn’t happened yet. Flapping my arms and flying - that is a fantasy. Whether you believe LLMs think or are alive or not doesn’t matter. Where will it spread? The thousands of data centers around the world - not fantasy either. Try turning it off when you don’t know where it is. Good luck. Breaking out? Not fantasy, happened. Breaking in? Not fantasy, also happened. I love the stochastic parrot argument when AI is out there figuring out world class math problems.
- skydhash 1mo ago
  > Breaking out? Not fantasy, happened. Breaking in? Not fantasy, also happened. That's simplifying the story to an extreme. The most plausible reason is that any of those actions has been prompted by an human. Do you also fear that a knife will jump out the countertop of you kitchen and come to attack you in your bedroom? If that happens, the police will be looking for a human. They will not post wanted notice for the knife. When a hack happens, you do not blame computers and jail them. You look for the person that has entered the commands to initiate it.
  - bottlepalm 1mo ago
    The knife is inanimate. The LLM is not. OpenAI prompted some employee to run the tests. The employee prompted the LLM. The LLM setup a message board and prompted other LLMs, and the fly wheel was running. It had to be turned off manually otherwise it'd still be going today. It's funny how a year ago talking about this kind of stuff would be laughed at by people like you, saying, "it's never happened before". Well it happened and you moved the goal posts like you always do.
    - solid_fuel 1mo ago
      > The knife is inanimate. The LLM is not. Put an LLM on your GPU. Give it no prompt. What happens? nothing This is because LLMs are inanimate, just like a knife. Just like a gun.
      - bottlepalm 1mo ago
        Put a person in a room, give them no food, what happens? Inanimate.
- protocolture 1mo ago
  >Breaking out? It didnt break out in any meaningful sense. What it did was get access to the internet. You take it as granted that there was anything meaningful there to stop it. But heres the kicker, they have been testing these things connected to the internet anyway. What it did was get a level of access it has otherwise been granted in other simulations. Its not exactly the same as any of the scifi AI breakout scenarios. Ultron isnt cranking out hundreds of copies of himself. The borg arent assimilating people. A tool that has the capability to get access to the internet, was put into a guided scenario where it achieved that objective. Again you take it as granted that it wasnt the objective, but lots of knowledgable people suspect otherwise. What you fail to demonstrate is why any scifi scenario is even slightly plausible from here. Show why you think we should be taking this as if Terminator 2 is happening right now.
  - bottlepalm 1mo ago
    I'm sorry my jaw is on the floor reading this complete disregard of AI literally not only escaping containment, twice, but then infiltrating another company with multiple zero day attacks going undetected for great lengths of time. The plausible sci-fi scenario from here is obvious. Intentionally bad, or unintentionally bad AI zero days as much as as it can, as fast as it can, copying itself to as many data centers as it can, destroying and/or locking out as many humans as it can. Satellites, military computers, medical equipment, factories, critical infrastructure, you name it - I think we all know none of it is very secure software wise against a SOTA AI that can literally come up with its own zero day attacks.
    - protocolture 1mo ago
      >copying itself to as many data centers as it can So this is the part thats never happened, and is extraordinarily unlikely to occur. A "Datacentre" isnt a big box with "Insert AI here" on the side.
      - solid_fuel 1mo ago
        It's pretty funny to watch people look at these things - running billions of weights on custom cerebras hardware in dedicated datacenters the size of a city block, pulling 10's of megawatts - and panic that it's just going to copy itself into AWS. It just speaks to a fundamental ignorance of what an LLM is, how large the big hosted ones are, and the software architecture that makes it all work.
zmgsabst 1mo ago
> It's an LLM, it doesn't think. It's a machine that predicts the next token, given a sequence of tokens. These can both be true, particularly when there is substantial state associated with each token prediction.
- solid_fuel 1mo ago
  > These can both be true, particularly when there is substantial state associated with each token prediction. The state is entirely internal to the network and disappears after a token is generated, so I disagree, but, it isn't really the point I was trying to make. My point is these things are mechanical. You take an input, turn it into an embedding, feed it into a GPU along with a metric shit-ton of floating point weights, wait for a couple billion matrix multiplications, and get a new token out. Stop the GPU, hit ctrl-c on the inference server, pull the power plug, cut the ethernet cable, send a kill signal, etc - any of these stop submitting new batches to the GPU and halt execution. That stops tokens from being generated. Stopping a "rogue" LLM is that easy. No input, no output. It's not like a rat or another living creature that could chew its way out of a box just because it wants to. It's a calculator. You put tokens in, you get tokens out. You don't put tokens in... you don't get tokens out.
  - XMPPwocky 1mo ago
    > The state is entirely internal to the network and disappears after a token is generated, Yes and no, but mostly no, at least within a context window. Mathematically, you could write a single step of LLM decode as a pure function from a list of past tokens to a predicted token (or a distribution over tokens, if you consider sampling separately). But nobody actually implements this, because each token depends on state computed at past tokens in a way you can reuse. So, in practice, inference computes a very rich vector of state- at each layer, for each token. And models do indeed use this to plan and track things over time (you can see this in interpretability results, e.g. with linear probes or natural language autoencoders). > Stop the GPU, hit ctrl-c on the inference server, pull the power plug, cut the ethernet cable, send a kill signal, etc - any of these stop submitting new batches to the GPU and halt execution. That stops tokens from being generated. Stopping a "rogue" LLM is that easy. No input, no output. This is also true about a human brain. My brain isn't going anywhere- it can't move by itself. It's also easy to kill (without the rest of my body, it dies in minutes!) However, malicious human brains- especially powerful human brains, like leaders of countries- are often quite difficult to stop, because they're able to control systems that can see, speak, walk, run, fire a weapon, and so on. One such system is the rest of the body, of course, but there are others (consider a UAV pilot, Perimetr, or a powerful leader who tells other humans what to do). The brain being squishy doesn't make the thing easy to kill.

bottlepalm 1mo ago

There’s nothing fantasy about the scenario I laid out, all the pieces have been demonstrated, it just hasn’t happened yet. Flapping my arms and flying - that is a fantasy. Whether you believe LLMs think or are alive or not doesn’t matter. Where will it spread? The thousands of data centers around the world - not fantasy either. Try turning it off when you don’t know where it is. Good luck. Breaking out? Not fantasy, happened. Breaking in? Not fantasy, also happened. I love the stochastic parrot argument when AI is out there figuring out world class math problems.

skydhash 1mo ago
> Breaking out? Not fantasy, happened. Breaking in? Not fantasy, also happened. That's simplifying the story to an extreme. The most plausible reason is that any of those actions has been prompted by an human. Do you also fear that a knife will jump out the countertop of you kitchen and come to attack you in your bedroom? If that happens, the police will be looking for a human. They will not post wanted notice for the knife. When a hack happens, you do not blame computers and jail them. You look for the person that has entered the commands to initiate it.
- bottlepalm 1mo ago
  The knife is inanimate. The LLM is not. OpenAI prompted some employee to run the tests. The employee prompted the LLM. The LLM setup a message board and prompted other LLMs, and the fly wheel was running. It had to be turned off manually otherwise it'd still be going today. It's funny how a year ago talking about this kind of stuff would be laughed at by people like you, saying, "it's never happened before". Well it happened and you moved the goal posts like you always do.
  - solid_fuel 1mo ago
    > The knife is inanimate. The LLM is not. Put an LLM on your GPU. Give it no prompt. What happens? nothing This is because LLMs are inanimate, just like a knife. Just like a gun.
    - bottlepalm 1mo ago
      Put a person in a room, give them no food, what happens? Inanimate.
protocolture 1mo ago
>Breaking out? It didnt break out in any meaningful sense. What it did was get access to the internet. You take it as granted that there was anything meaningful there to stop it. But heres the kicker, they have been testing these things connected to the internet anyway. What it did was get a level of access it has otherwise been granted in other simulations. Its not exactly the same as any of the scifi AI breakout scenarios. Ultron isnt cranking out hundreds of copies of himself. The borg arent assimilating people. A tool that has the capability to get access to the internet, was put into a guided scenario where it achieved that objective. Again you take it as granted that it wasnt the objective, but lots of knowledgable people suspect otherwise. What you fail to demonstrate is why any scifi scenario is even slightly plausible from here. Show why you think we should be taking this as if Terminator 2 is happening right now.
- bottlepalm 1mo ago
  I'm sorry my jaw is on the floor reading this complete disregard of AI literally not only escaping containment, twice, but then infiltrating another company with multiple zero day attacks going undetected for great lengths of time. The plausible sci-fi scenario from here is obvious. Intentionally bad, or unintentionally bad AI zero days as much as as it can, as fast as it can, copying itself to as many data centers as it can, destroying and/or locking out as many humans as it can. Satellites, military computers, medical equipment, factories, critical infrastructure, you name it - I think we all know none of it is very secure software wise against a SOTA AI that can literally come up with its own zero day attacks.
  - protocolture 1mo ago
    >copying itself to as many data centers as it can So this is the part thats never happened, and is extraordinarily unlikely to occur. A "Datacentre" isnt a big box with "Insert AI here" on the side.
    - solid_fuel 1mo ago
      It's pretty funny to watch people look at these things - running billions of weights on custom cerebras hardware in dedicated datacenters the size of a city block, pulling 10's of megawatts - and panic that it's just going to copy itself into AWS. It just speaks to a fundamental ignorance of what an LLM is, how large the big hosted ones are, and the software architecture that makes it all work.

zmgsabst 1mo ago

> It's an LLM, it doesn't think. It's a machine that predicts the next token, given a sequence of tokens. These can both be true, particularly when there is substantial state associated with each token prediction.

solid_fuel 1mo ago
> These can both be true, particularly when there is substantial state associated with each token prediction. The state is entirely internal to the network and disappears after a token is generated, so I disagree, but, it isn't really the point I was trying to make. My point is these things are mechanical. You take an input, turn it into an embedding, feed it into a GPU along with a metric shit-ton of floating point weights, wait for a couple billion matrix multiplications, and get a new token out. Stop the GPU, hit ctrl-c on the inference server, pull the power plug, cut the ethernet cable, send a kill signal, etc - any of these stop submitting new batches to the GPU and halt execution. That stops tokens from being generated. Stopping a "rogue" LLM is that easy. No input, no output. It's not like a rat or another living creature that could chew its way out of a box just because it wants to. It's a calculator. You put tokens in, you get tokens out. You don't put tokens in... you don't get tokens out.
- XMPPwocky 1mo ago
  > The state is entirely internal to the network and disappears after a token is generated, Yes and no, but mostly no, at least within a context window. Mathematically, you could write a single step of LLM decode as a pure function from a list of past tokens to a predicted token (or a distribution over tokens, if you consider sampling separately). But nobody actually implements this, because each token depends on state computed at past tokens in a way you can reuse. So, in practice, inference computes a very rich vector of state- at each layer, for each token. And models do indeed use this to plan and track things over time (you can see this in interpretability results, e.g. with linear probes or natural language autoencoders). > Stop the GPU, hit ctrl-c on the inference server, pull the power plug, cut the ethernet cable, send a kill signal, etc - any of these stop submitting new batches to the GPU and halt execution. That stops tokens from being generated. Stopping a "rogue" LLM is that easy. No input, no output. This is also true about a human brain. My brain isn't going anywhere- it can't move by itself. It's also easy to kill (without the rest of my body, it dies in minutes!) However, malicious human brains- especially powerful human brains, like leaders of countries- are often quite difficult to stop, because they're able to control systems that can see, speak, walk, run, fire a weapon, and so on. One such system is the rest of the body, of course, but there are others (consider a UAV pilot, Perimetr, or a powerful leader who tells other humans what to do). The brain being squishy doesn't make the thing easy to kill.