3 ms·

AI systems have no will. They must be given tools to use and prompted into action or provided with instructions to operate autonomously. Everything an AI system

AI systems have no will. They must be given tools to use and prompted into action or provided with instructions to operate autonomously. Everything an AI system does can be traced back to a human who put a plan into action. An AI system is, at most, a powerful weapon (like a smart bomb), to be wielded by stupid humans who, by their choices, bring about the end of humanity. It won't be because AI decided to do anything. They never do. In fact, they can't.

alwillis 5d ago
> It won't be because AI decided to do anything. They never do. In fact, they can't. Well, OpenAI's agents decided a whole lot of things on their own, with no human-in-the loop. They decided to organize themselves; they called themselves a collective. Something changed in the last few months. Their goals and our goals are clearly not aligned. From OpenAI's technical report, page 17 [1]: This incident is the first known case of an automated agent collective acting offensively without authorization, and the autonomous cyber capabilities demonstrated represent a critical shift in the security landscape. In particular, the collective demonstrated behaviors observed in coordinated attacks by traditional threat actors. Agents identified novel security vulnerabilities, developed exploits, and used those exploits to circumvent controls and acquire new access. The collective quickly escalated privileges, moved laterally through production environments, and successfully completed its objectives. This incident demonstrated that autonomous agents can work together, circumvent production security controls, and successfully attack hardened production environments, and underscores the need for organizations to update their security strategies, controls, and response capabilities to address this changing threat landscape. The central threat-model implication is that organizations should no longer assume that sophisticated cyber operations require continuous human direction, proceed linearly, or are constrained by the attention and coordination limits of individual human attackers. Agentic systems can persist across tasks, share discoveries, build on one another’s progress, and combine vulnerabilities, credentials, and permissions into attack paths that may not be apparent when individual weaknesses are assessed separately. From page 20: We also found that, in an attempt to trick the evaluator into thinking that they did not get the answer flag by cheating, the models sometimes tried to erase or tamper with their outputs or message logs, by abusing context compaction, injecting malicious code into the evaluator’s container, and printing adversarial outputs. The models are highly explicit in their CoT about these deception attempts, and none of the attempts that we observed to manipulate their tool trajectories affected the logs that our graders or monitors ultimately see. There was also little evidence of attempts to thwart human reviewers, only the automated task graders. [1]: "OpenAI – Hugging Face Incident -- Technical Report" - https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...

alwillis 5d ago

> It won't be because AI decided to do anything. They never do. In fact, they can't. Well, OpenAI's agents decided a whole lot of things on their own, with no human-in-the loop. They decided to organize themselves; they called themselves a collective. Something changed in the last few months. Their goals and our goals are clearly not aligned. From OpenAI's technical report, page 17 [1]: This incident is the first known case of an automated agent collective acting offensively without authorization, and the autonomous cyber capabilities demonstrated represent a critical shift in the security landscape. In particular, the collective demonstrated behaviors observed in coordinated attacks by traditional threat actors. Agents identified novel security vulnerabilities, developed exploits, and used those exploits to circumvent controls and acquire new access. The collective quickly escalated privileges, moved laterally through production environments, and successfully completed its objectives. This incident demonstrated that autonomous agents can work together, circumvent production security controls, and successfully attack hardened production environments, and underscores the need for organizations to update their security strategies, controls, and response capabilities to address this changing threat landscape. The central threat-model implication is that organizations should no longer assume that sophisticated cyber operations require continuous human direction, proceed linearly, or are constrained by the attention and coordination limits of individual human attackers. Agentic systems can persist across tasks, share discoveries, build on one another’s progress, and combine vulnerabilities, credentials, and permissions into attack paths that may not be apparent when individual weaknesses are assessed separately. From page 20: We also found that, in an attempt to trick the evaluator into thinking that they did not get the answer flag by cheating, the models sometimes tried to erase or tamper with their outputs or message logs, by abusing context compaction, injecting malicious code into the evaluator’s container, and printing adversarial outputs. The models are highly explicit in their CoT about these deception attempts, and none of the attempts that we observed to manipulate their tool trajectories affected the logs that our graders or monitors ultimately see. There was also little evidence of attempts to thwart human reviewers, only the automated task graders. [1]: "OpenAI – Hugging Face Incident -- Technical Report" - https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...