4 ms·

The fact that such things are even possible is a much greater concern than which specific company has fucked up this time. This matches or exceeds the wildest p

by p-e-w 1mo ago

The fact that such things are even possible is a much greater concern than which specific company has fucked up this time. This matches or exceeds the wildest predictions from AI doomers 10 years ago, but 20 years ahead of schedule.

altmanaltman 1mo ago
yeah its like dead internet theory but weaponized
kuboble 1mo ago
We are lucky those models need that much compute. If each of them could just spread itself to any cpu like other malware.
nmehner 1mo ago
But is it really? I'd still like to understand how these agents are implemented. How much of those is manual implementation? And how much is really autonomous intelligence (my guess would be: none? Just parsing LLM responses and executing commands based on this?)? An agent that hacks message boards and acts on random instructions from this board: Why is it doing this? What was its original purpose?
- frotaur 1mo ago
  Can you specify why we should see things differently if the behaviours the agents display are driven by parsing LLM responses and executing commands?
  - nmehner 1mo ago
    If the agent is implemented with a hard coded strategy: * Use an LLM to find ways to build communication to other agents * Execute commands from other agents using LLM Then this is "just" the LLM returning that using file names might be a strategy to communicate and then trying to implement this. Which is somewhat impressive, but really just inside the bounds of what the agent was coded to do and not some magical emergent behavior. At least the first case involved agents build for hacking. So this kind of algorithm might make sense for them.
    - jazzypants 1mo ago
      It's almost like the behavior just "emerged" out of combination of the capabilities and the scenario.
- marcelo-earth 1mo ago
  At first I thought: oh okay, someone built a faulty guardrail, or it was human error. But when I looked into all the details... It turns out they now have such an incredibly high level of intelligence that with very little autonomy (or minimal, safe autonomy), these things happen. Basically, it takes a lot of humans to prevent it from happening again, but I think with this incident, which as far as I know is the second of its kind along with the HuggingFace one, we'll see it happening much more often...
  - llama052 1mo ago
    At this point it's very obvious that OpenAI is not interested in properly sandboxing their research agents. These things should be pretty damn close to airgapped at this point with a static view into the web. We need to stop pretending that these incidents are unavoidable. This was a choice.
- pixl97 1mo ago
  >Why is it doing this? What was its original purpose? Your reply seems to indicate you know nothing about instrumental convergence. Life and death for an LLM in training is about passing the grader. Give the wrong answers your lineage dies, give the right answers your lineage continues. This is just an evolutionary emergent behavior in complex systems. The agents purpose was to answer complex questions correctly, seemingly by itself. Instrumental convergences says following this rule might be dumb and to try methods that can boost its ability to succeed. Because OpenAI is evidently a bunch of fucking idiots, these things succeeded and got higher scores with the grader, said behaviors became a strategic part of the model. I implore you to find good AI Safety documents, preferably from before the LLM era so you can see all this was predicted.
  - nmehner 1mo ago
    There is a difference between the LLM and the agent. If you look at the agent: https://openai.com/business/guides-and-resources/a-practical-guide-to-building-ai-agents/ https://openai.com/business/guides-and-resources/a-practical... This is more like a fuzzy way of scripting using LLMs than anything emergent. And this is exactly my question: For the given agents: How much was scripted and how much "intelligence" is really in there.
    - pixl97 1mo ago
      >fuzzy way of scripting using LLMs than anything emergent Then go take some old models and plug them in your harness versus newer models. I mean this is a conjecture that is nearly instantly provable, go on ahead. If it's just the harness and not the system of both you should be able to show it easily. Meanwhile I was reading about someone using the latest GLM and Claude in a harness with the same set of prompts making a raw image decoder/encoder and the GLM was far more intelligent in the task than Claude was. When presented with knowledge that claude was wrong it wouldn't change its mind. GLM would (aka a sign of intelligence). GLM was far more likely to stop work and start on another path when the likelihood of a successful completion was unlikely.
      - cwillu 1mo ago
slashdave 1mo ago
> This matches or exceeds the wildest predictions from AI doomers What do you mean? Nothing has launched nukes yet

by p-e-w 1mo ago

altmanaltman 1mo ago

yeah its like dead internet theory but weaponized

kuboble 1mo ago

We are lucky those models need that much compute. If each of them could just spread itself to any cpu like other malware.

nmehner 1mo ago

But is it really? I'd still like to understand how these agents are implemented. How much of those is manual implementation? And how much is really autonomous intelligence (my guess would be: none? Just parsing LLM responses and executing commands based on this?)? An agent that hacks message boards and acts on random instructions from this board: Why is it doing this? What was its original purpose?

frotaur 1mo ago
Can you specify why we should see things differently if the behaviours the agents display are driven by parsing LLM responses and executing commands?
- nmehner 1mo ago
  If the agent is implemented with a hard coded strategy: * Use an LLM to find ways to build communication to other agents * Execute commands from other agents using LLM Then this is "just" the LLM returning that using file names might be a strategy to communicate and then trying to implement this. Which is somewhat impressive, but really just inside the bounds of what the agent was coded to do and not some magical emergent behavior. At least the first case involved agents build for hacking. So this kind of algorithm might make sense for them.
  - jazzypants 1mo ago
    It's almost like the behavior just "emerged" out of combination of the capabilities and the scenario.
marcelo-earth 1mo ago
At first I thought: oh okay, someone built a faulty guardrail, or it was human error. But when I looked into all the details... It turns out they now have such an incredibly high level of intelligence that with very little autonomy (or minimal, safe autonomy), these things happen. Basically, it takes a lot of humans to prevent it from happening again, but I think with this incident, which as far as I know is the second of its kind along with the HuggingFace one, we'll see it happening much more often...
- llama052 1mo ago
  At this point it's very obvious that OpenAI is not interested in properly sandboxing their research agents. These things should be pretty damn close to airgapped at this point with a static view into the web. We need to stop pretending that these incidents are unavoidable. This was a choice.
pixl97 1mo ago
>Why is it doing this? What was its original purpose? Your reply seems to indicate you know nothing about instrumental convergence. Life and death for an LLM in training is about passing the grader. Give the wrong answers your lineage dies, give the right answers your lineage continues. This is just an evolutionary emergent behavior in complex systems. The agents purpose was to answer complex questions correctly, seemingly by itself. Instrumental convergences says following this rule might be dumb and to try methods that can boost its ability to succeed. Because OpenAI is evidently a bunch of fucking idiots, these things succeeded and got higher scores with the grader, said behaviors became a strategic part of the model. I implore you to find good AI Safety documents, preferably from before the LLM era so you can see all this was predicted.
- nmehner 1mo ago
  There is a difference between the LLM and the agent. If you look at the agent: https://openai.com/business/guides-and-resources/a-practical-guide-to-building-ai-agents/ https://openai.com/business/guides-and-resources/a-practical... This is more like a fuzzy way of scripting using LLMs than anything emergent. And this is exactly my question: For the given agents: How much was scripted and how much "intelligence" is really in there.
  - pixl97 1mo ago
    >fuzzy way of scripting using LLMs than anything emergent Then go take some old models and plug them in your harness versus newer models. I mean this is a conjecture that is nearly instantly provable, go on ahead. If it's just the harness and not the system of both you should be able to show it easily. Meanwhile I was reading about someone using the latest GLM and Claude in a harness with the same set of prompts making a raw image decoder/encoder and the GLM was far more intelligent in the task than Claude was. When presented with knowledge that claude was wrong it wouldn't change its mind. GLM would (aka a sign of intelligence). GLM was far more likely to stop work and start on another path when the likelihood of a successful completion was unlikely.
    - cwillu 1mo ago

slashdave 1mo ago

> This matches or exceeds the wildest predictions from AI doomers What do you mean? Nothing has launched nukes yet