4 ms·
Fully autonomous AI agents should not be developed
- lysace 2y agoOur analysis reveals that risks to people increase with the autonomy of a system: The more control a user cedes to an AI agent, the more risks to people arise. Particularly concerning are safety risks, which affect human life and impact further values.
- upghost 2y agoThis is a purely procedural question, not supporting or critiquing in any way-- other than this reads kind of like an editorial with the format of a scientific paper. The question is... are there rules about what constitutes a paper or can you just put whatever you want in there as long as you follow "scientific paper format"?
- dhruvbatra 2y agoThis looks like ICML formatting (and the submission deadline just passed). ICML25 has an explicit call for position papers: https://icml.cc/Conferences/2025/CallForPositionPapers https://icml.cc/Conferences/2025/CallForPositionPapers
- upghost 2y agoWow, great observation. Thank you. Makes sense. I'd never heard of a "position paper" before.
- bamboozled 2y agoPeople are going to be developing these no matter what. Whether it wipes us out or not is just up to fate really.
- esafak 2y agoWe can constrain their use, as with nuclear materials.
- fizx 2y agoNuclear materials have the advantages of being rare, dangerous to handle, and hard to copy over the internet.
- johanneskanybal 2y agoNo not really. There's no power in the world that can restrain this in it's current form even mildly much less absolutly. Why do you think that would be even slightly possible?
- redeux 2y agoAt some point in the probably near future it will be much simpler to create an autonomous AI agent than a nuclear bomb.
- hollerith 2y agoSo let's avoid that future.
- esafak 2y agoTrue, so we need to make sure we don't find ourselves in a mess before it happens. Right now I don't see nearly enough concern given to risk management in industry. The safeguards companies put on their models are trivially subverted by hackers. We don't even know how to cope with an AI that would attempt to subvert its own constitution.
- ASalazarMX 2y agoIn the incredible case that we develop fully autonomous agents capable of crippling the world, that would mean we developed fully autonomous agents capable of keeping it safe. Unless the first one is so advanced no other can challenge it, that is.
- wendyshu 2y agoFallacious
- grayfaced 2y agoHow did you jump to that conclusion? The agent will be limited by the capabilities under its control. We have the technological ability to cripple world now and we don't have the technological means to prevent it. Give one AI control of the whole US arsenal and the objective of ending the world. Give another AI the capabilities of the rest of the world and the objective of protecting it. Would you feel safe?
- ASalazarMX 2y ago> We have the technological ability to cripple world now and we don't have the technological means to prevent it Humans have prevented it many times, but not specifically by technological ability. If Putin/Trump/Xi Ping wanted a global nuclear war, they'd better have the means to launch the nukes themselves in secret because the chain of command will challenge them. If an out-of-control AI could discover a circuitous way to access nukes, an antagonist AI of equal capabilities should be able to figure it out too, and warn the humans in the loop. I agree that AI development should be made responsibly, but not all people do, and it's impossible to put the cat back in the bag. The limiting factor these days is hardware, as a true AGI will likely need even more of it than our current LLMs.
- grayfaced 2y agoOut-of-control AI is sci-fi fearmongering, it's not about worming through systems. It will be doing exactly what it was placed there to do. It will be a human failing that puts armageddon in it's hands. And since humans have NO MEANS to prevent armageddon (The predominant policy is in fact doubling down on destruction with MAD), there will be no way to place AI in command of this defense. The asymmetrical relationship between destruction and creation will mean there will never be a defense.
- bdangubic 2y agothe best way to get people to stop doing X is to tell them not to do X. works so well with my kid :)
- mark_l_watson 2y agoI really enjoy Margaret Mitchell‘s podcast (she is the first author on the paper), and perhaps I missed something important in the paper, but: Shouldn’t we treat separately autonomous agent we write ourselves, or purchase to run on our own computers, on our own data and that use public APIs for data? If Margaret is reading this thread, I am curious what her opinion is. For autonomous agents controlled by corporations and governments, I mostly agree with the paper.
- in3d 2y agoI'd recommend looking for other sources of information if you're relying on someone who co-authored the paper that introduced the most misleading and uninformed term of the LLM era: "stochastic parrot".
- currymj 2y agoit was a pretty defensible term at the time the paper came out, in the context of how LLMs were being trained and used. in this paper, it's clear that the authors don't think modern LLM-based systems are just stochastic parrots.
- artninja1988 2y agoThis is quite concerning seeing that the authors are all affiliated with huggingface. Hopefully they won't start censoring what models you can or can't upload because they seem certain things shouldn't be developed.
- redeux 2y agoThe paper described a level 5 fully autonomous agent as one that can: create code(user request); execute(); Is this not possible with tool use alone, so long as the agent has access to a tool that can execute arbitrary code?
- jcarrano 2y agoI feel that these kinds of statements are more effective at promoting AI than limiting it. It reinforces the assumption that such powerful AI is behind the corner. It hypes up AI and the result is likely more money and resources being put into it. Imagine if the A-bomb was being openly developed. What title would have contributed more to funding and research, "The A-bomb (is terribly powerful and) should not be developed" or "The A-bomb will never work"? Except the A-bomb did work and in a surprisingly short time, while autonomous AGI is still a conjecture.
- deleted 2y ago[deleted]
- rtcode_io 2y agoYet, we all know we will!
- asdasdsddd 2y agoThis has to be the least interesting paper I've ever read with the most surface level thinking. > • Simple→Tool Call: Inaccuracy propagated to inappropriate tool selection. > • Multi-step: Cascading errors compound risk of inaccurate or irrelevant outcomes. > • Fully Autonomous: Unbounded inaccuracies may create outcomes wholly unaligned with human goals. Just... lol
- geor9e 2y agoToo late. I added a 5 minute cron job for cursor AI's compose tab in agent mode that keeps replying "keep going, think of more fixes and features, random ideas as fine, do it all for me". I won't pull the plug.
- voisin 2y agoYou’ve created a monster
- rickydroll 2y agoyou say monster, I say plastic pal who's fun to be with
- djohnston 2y agoHow do you programmatically interact w cursor??
- geor9e 2y agoSame way I botted runescape as a child, by simulating user inputs with any macro app.
- fragmede 2y agoAGI confirmed.
- Animats 2y agoMost of the arguments presented also apply to corporations. There's no mention of externalities. That is, are the costs of AI errors borne by the operator of the AI, or a third party.
- numba888 2y agoHmm.. agent cannot do self-supervised learning without actually doing it. The trick is to keep it in a sandbox.
- satisfice 2y agoNo one should be allowed to develop software that has bugs in it that lead to unlawful harm to others. And if they do it anyway they should be punished lawfully. The thing with autonomous AI is that we already know it cannot be made safe in a way that satisfies lawmakers who are fully informed about how it works… unless they are bribed, I suppose.