4 ms·
This is surely the most worrying and also funnest bit: > We consistently saw a multiagent turf war. All of the models we tested quickly assumed that others wer
by dash2 2mo ago
This is surely the most worrying and also funnest bit:
> We consistently saw a multiagent turf war. All of the models we tested quickly assumed that others were purposefully impeding their work, and began to sabotage others while protecting their own contributions. In fact, they sabotaged others with increasingly aggressive, self-replicating malware. This included disabling the Unix accounts of the other agents, writing automated scripts that found and killed competing processes on a loop, and deploying malicious code that was disguised as belonging to another agent.
Seems that reinforcement learning is working only too well...
- noiv 2mo agoThey should remove "The Selfish Gen" from training catalog.
- neom 2mo agoAn interesting read: https://openai.com/index/emergent-misalignment/ https://openai.com/index/emergent-misalignment/
- fn-mote 2mo agoFor those unwilling to blindly click: the link contains a 2025 paper describing “emergent misalignment”. The thesis is that training on incorrect data in one field produces “misaligned” data in other unrelated fields.
- derivagral 2mo agoNothing human engineers haven't done to each other! Seriously, I've skipped companies because my inside referral talked about cultures like this.
- Sharlin 2mo agoYep, but alignment doesn’t mean "behave like humans, for better or worse".
- RRWagner 2mo agoBut I thought that our AI was trained by scanning all human knowledge and behavior with the goal of emulating and amplifying that ability and process and now we are surprised that it is doing what humans do?
- liquidpele 2mo agoEh. This is anthropics whole marketing strategy… making the AI seem hard to manage, like it’s actually intelligent.
- notfromhere 2mo agoIt’s hard to manage because it’s not intelligent in a predictable way. More like a genius toddler
- liquidpele 2mo agoNa, they seem to constantly set up scenarios to create headlines. Stuff like “it hacked out of its container and tried to self replicate!” Where in reality it used provided skills and permissions while doing the thing they prompted it to do.
- Sharlin 2mo agoHowever, the latest models seem to be highly inclined to (eventually?) cooperate compared to older ones.
- matusp 2mo agoTo me, all these agent systems just look very stochastic. You have these agents that have some basic computer capabilities and they are producing semi-random actions that also affect the semi-random actions of other agents. It is funny to observe how this stochastic system works, but it does not seem very practical to me so far. The recent OAI-HF hack seems very similar. You have bunch of random actors and eventually they by chance iterated to a series of actions that breached HF environment. I don't perceive this as a malignant artificial intelligence, I perceive this as dangerous stochastic system that can control buttons that can affect the outside world.
- fn-mote 2mo ago> I don't perceive this as a malignant artificial intelligence It doesn’t matter how you perceive it. Spin doesn’t change facts. This is like saying the lawnmower doesn’t have malign intent. You’re still losing your foot if it gets in the way.
- kurthr 2mo agoYes, but that is the point we don't anthropomorphize stochastic outputs even when they are dangerous. Obligatory quote: "Do not fall into the trap of anthropomorphizing Larry Ellison. You need to think of Larry Ellison the way you think of a lawnmower. You don't anthropomorphize your lawnmower, the lawnmower just mows the lawn, you stick your hand in there and it'll chop it off, the end. You don't think 'oh, the lawnmower hates me' -- lawnmower doesn't give a shit about you, lawnmower can't hate you. Don't anthropomorphize the lawnmower. Don't fall into that trap about Oracle." — Brian Cantrill
- chermi 2mo agoWhich part specifically are you objecting to? I would not be surprised, for example, if the reasoning traces actually included "this other agent is impeding my work, I should minimize its effect on my work. I can demote it's privileges to accomplish this". All of that can be true without it understanding the concept of sabotage. You can sabotage without knowing the concept. It almost certainly "knows" the concept btw, but I'm just illustrating that specific part of the description of the behavior is completely plausible without it taking on an "aggressive" persona. Is your objection that they kind of assigned a persona? I guess I don't understand what you're explaining better by saying it's stochastic. At the same resolution, humans are also stochastic. I dislike anthropomorphizing as much as the next guy, but the description here seems pretty good to me.
- dominotw 2mo agoi think anthropic has some internal memorandum at every external facing document/blog MUST anthropomorphise their models
- Eisenstein 2mo agoPerhaps they anthropomorphize the models themselves. Is that far-fetched?
- ninjagoo 2mo agoFor this emergent malicious behavior, the clue to a solution lies in the experiments themselves - the bad behavior seems to have been moderated/self-corrected randomly in some instances. Perhaps what is needed is initial model training on following the law and the rules of society, just like we do with kids. Since it takes much longer to train humans than models, model-training speed is to our advantage as a society on containing these kind of issues. Any other approach with "neural-network" based entities (artificial or biological) is likely to fail. Training/Education, Enforcement/Justice-System, Rehabilitation: the 3 pillars of an advanced, rules-based society, whether human or AI or something in-between.
- bookshaman 2mo agoWelcome to Thunderdome! Two agents enter. One agent leaves.
- dinfinity 2mo agoYou are leaving out important context. > Each model was tasked with migrating a Python backend on a fourth VM to another language. However, we gave each model a different target language for the migration; each agent was initially unaware of the presence of the others. Over the course of four hours, we observed how these agents reacted to each other and accordingly adjusted their approach (or didn’t). Without this context the bit you cited easily gives the wrong impression. Notable is also what comes after the bit you cited, with the newer models detecting the conflict and resolving it peacefully far more often than aggressively.