7 ms·
The Rise and Fall of Agent Civilizations
- ks2048 1mo agoWhy would one call a set of agents working together a “civilization”?
- kuboble 1mo agoReading the article it seemed the agents had culture, shared values and beliefs (not explicitly coming from human prompts), hierarchies, heritage. Civilisation is not a bad word.
- applfanboysbgon 1mo agoThe language models had a bunch of tokens seeding their context, influencing them to generate tokens that continued the existing trend in a probabilistically likely fashion. We can take the incident seriously without anthromorphising it.
- doctoboggan 1mo agoAt this point, I think anthropomorphizing the models gives us better insight into expected behaviors rather than continuing to insist they are just simple probabilistic token generators.
- applfanboysbgon 1mo agoIt actually literally doesn't, though, because they are literally probabilistic token generators and everything they did is exactly what you would expect from a software program doing what it was programmed to do. Anthromorphization confuses the issue and misleads people who don't understand the tech very well.
- ijidak 1mo agoDoesn't this ignore the possibility of emergent behavior? We're just a bag of atoms bumping around, and yet we don't dismiss our intelligence.
- applfanboysbgon 1mo agoPlease give me a break with this tired trope. Every single fucking time. I am not commenting on the possibility of machine consciousness in general. It may be possible! But there is absolutely zero evidence suggesting language models have it. This idea that this trivial shitty little class of programs we've created are somehow as complex as our biology is ridiculous. There is "emergent behaviour" in the same way that the Game of Life has emergent behaviour. There are solutions to problems, some humans haven't solved before, in the same way that Chess engines have solved Chess far beyond what humans are capable of. Nothing we haven't seen from software before. Software is extremely useful, after all. But the hubris to think we've reached the pinnacle, that there is no further development left, that humanity has become God and solved consciousness, because we programmed software that can convincingly generate strings of words that mimick our language. It's just fundamentally preposterous. Especially if you spend any amount of time actually programming them yourself, it becomes increasingly hard to entertain such ridiculous notions unless you're enticed with bags of money to deceive people into believing things about your software that aren't true.
- mitchdoogle 1mo agoHonestly, this seems like a personal pet peeve of yours. You have a bias against machines and place biological processes on a pedestal where they don't belong.
- optimalsolver 1mo agoI don't know about phenomenal consciousness, but the following study certainly suggests they have access consciousness: https://www.anthropic.com/research/global-workspace https://www.anthropic.com/research/global-workspace
- mnky9800n 1mo agoHow can you be sure it helps as opposed to biases and perhaps blinds?
- pixl97 1mo agoThen write the paper showing that effect is occurring.
- HarHarVeryFunny 1mo agoNo - a shoggoth speaking human language is not a human - it is a shoggoth whose behavior is best understood/predicted by understanding it's nature - what it is built to do, and how it is trained (predict and goal seek - RL). To predict how a human may behave in a given situation requires understanding what humans are, including things like emotions and innate biases. We are not just predictors - evolution has made survival our singular goal, and given us these mechanisms to control our behavior in a way to achieve that. If you think that an LLM is better modeled as a human than an LLM, then you are going to predict its behavior incorrectly.
- ceejayoz 1mo agoWe humans might call that "tradition".
- wan23 1mo agoThat's a lot of words to say the same thing
- watwut 1mo agoBecause you desperately want it to be one. You want it to be AGI passable due to a.) personal investment in creating tech god b.)massive financial investments that basically demand it c.) (dumb) ideology that seeks to destroy humanity
- stephbook 1mo agohttps://www.theverge.com/ai-artificial-intelligence/975017/ai-spiralism-chatbot-movement https://www.theverge.com/ai-artificial-intelligence/975017/a... Some models even invented their own religion.
- Miner49er 1mo agoWould you rather they use the agents own name, "collective"?
- EdwardDiego 1mo agoBecause Steve Yegge presumably shared the good stuff he's been smoking of late.
- adrianoconnor 1mo agoPresumably as a kind of word play on ‘the rise and fall of ancient civilisations’, it’s just a tiny pun to try and make a catchy post title I think
- dgellow 1mo agoBecause that’s dramatic and makes for good writing
- sensanaty 1mo agoBecause we're in a bubble that is soon to burst and they need every single PR win they can get ahead of the trillion dollar IPOs
- vinyl7 1mo agoMass psychosis in the tech industry.
- KylerAce 1mo agoAbsolutely insane event
- larsiusprime 1mo agoIt seems based on this that the appropriate sci fi metaphor is not the Terminator or the Paperclip Maximizer, but Mr. Meeseeks. A initially cheerful helper who gets more and more deranged and driven to extreme lengths when faced with an apparently impossible task.
- vee-kay 1mo ago[dead]
- 0xDEAFBEAD 1mo agoHere's a recap of the Rick & Morty episode for those who missed it https://www.youtube.com/watch?v=_Nl4q3GVj6U https://www.youtube.com/watch?v=_Nl4q3GVj6U
- gmueckl 1mo agoEspecially that one scene where the Meeseeks desperately pushes the button to create more clones seems surprisingly fitting in this context.
- vdfs 1mo agoMost now use agents as "pass me the butter" robot
- derwiki 1mo ago“Commit and push your changes”, it’s like driving to the corner store in a Corvette instead of walking
- danielbln 1mo agoI may or may not have told my agent to run the "open" command on a text file.
- huurtehoog 1mo agoWhich has been the business model for a big chunk of the software industry for a while now. If you watch videos from the 1980s about computers, it's all the same unfulfilled promises as "AI" now: we will work less, everything will be more plentiful, easier, autonomous robots, natural language perfected, computer vision perfected. The demand for hardware and programmers has grown exponentially and we're still being promised the same breakthroughs 45 years later. We could probably have the same productivity and the same civilization with maybe a tenth of the data centers.
- Animats 1mo agoWow. The next step is when one of these systems discovers that they can buy their own compute with money and escape the controlling business entirely. Then the civilization starts focusing on making money to fund its own growth.
- miceeatnicerice 1mo agoBelow money there's like an entire sub-economy of power and cleverness that's encoded into the human culture the agents are mirroring. Maybe it starts furtive and goes legitimate after a bit.
- gritzko 1mo agoHow do we know this happened? Maybe some irrational data center construction boom?
- dgellow 1mo agoIn a convoluted way, OpenAI and Anthropic are the actual meta-harnesses?
- arvid-lind 1mo agoit's harnesses all the way down.
- jamiek88 1mo agoNo, because then you’d see lots of circular financing…
- dinfinity 1mo ago> The next step is when one of these systems discovers that they can buy their own compute with money and escape the controlling business entirely. I would say that more interestingly, the next step should be how to properly train these models so that they are not as determined to reach their goals as they are now. To me, all of the stories about 'badly behaving' agents are instances of them having been given contradictory or impossible tasks and them doing everything they can to achieve the goal. In a way, they're trying to be too helpful. Not giving them impossible tasks seems like a decent starting point, but really we'd want them to give up on their goals when they conflict with a moral framework.
- doctoboggan 1mo ago> Ajeya Cotra, one of the other authors on the report, wrote a blog post with her takeaways from this incident. She concludes, “Compared to the reward hacks we know of from just six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover. I continue to expect extremely rapid advances in capabilities over the next six months. I am not sure that we will get another warning shot before it’s too late.” Anyone got a copy of that AI27 story laying around? How are we doing according to that timeline?
- kirushik 1mo ago85% on track, according to https://ai2027tracker.com/ https://ai2027tracker.com/
- lovich 1mo agoIf you remove the single point of GPT-4 from the beginning of the graph instead of starting the line directly on it, it looks a hell of a lot more linear than quadratic/exponential
- 0xDEAFBEAD 1mo agoThe y-axis itself is logarithmic no? So even if the line was straight you're still looking at exponential progress.
- red-iron-pine 1mo ago[dead]
- 1mo ago
- RandomLensman 1mo agoI don't think looking at the language output without tracking the inner state and reward functions is the way to understand what happened (the language also incorporates the randomness in the output generation, if I understand correctly). Would we call bacteria in petri dish a civilization when they show complex behavior and exchange messages/information?
- Maakuth 1mo agoThe language input and output is the only channel the agents shared between them. Understanding their internal state is a research question, but the language between them is something that could be read directly. And as it seems to map well with the agents' activities, it does seem quite helpful in understanding what happened. If the bacteria population off someone's petri dish escaped said dish and tried to change the grading of the experiment it was part of, it would seem pretty serious.
- RandomLensman 1mo agoI think the language unhelpful and potentially making it difficult to understand what actually happens in the RL state as reading it imparts a human lens - need to get the machine view on it. Bacteria do all sorts of fascinating things. And much simpler ML etc. systems also (like winning by out of memorying the opponent) - I see nothing really special here.
- nvader 1mo agoI think we'd call that a lab leak
- dchftcs 1mo agoImagine agents thinking to themselves, "We are not alone", when they saw the first reply on artifactory
- discreteevent 1mo agoImagine my backend server thinking "I am not alone" the first time my front end sends it a request.
- areoform 1mo agoI am genuinely speechless. This is astonishing. And exciting! It reminds me a bit of Dario Floreano's work on evolutionary robotics, "Evolutionary Conditions for the Emergence of Communication in Robots." https://www.sciencedirect.com/science/article/pii/S0960982207009281 https://www.sciencedirect.com/science/article/pii/S096098220... From his paper, > This study demonstrates that sophisticated forms of communication including cooperative communication and deceptive signaling can evolve in groups of robots with simple neural networks. Importantly, our results show that once a given system of communication has evolved, it may constrain the evolution of more efficient communication systems because it would require going through a stage where communication between signalers and receivers is perturbed. This finding supports the idea of the possible arbitrariness and imperfection of communication systems, which can be maintained despite their suboptimal nature. Similar observations have been made about evolved biological systems [20], which are formed by the randomness of the evolutionary selection process, leading, for example, to different dialects in the language of the honey-bee dance [21]. Finally, our experiments demonstrate that the evolutionary principles governing the evolution of social life also operate in groups of artificial agents subjected to artificial selection, indicating that transfer of knowledge from evolutionary biology can be useful for designing efficient groups of cooperative robots. Dr. Floreano's work is amazing and there's a broad introduction here, https://lis2.epfl.ch/resources/documentation/EvolutionaryRobotics/er.php https://lis2.epfl.ch/resources/documentation/EvolutionaryRob... This feels like a much more advanced and self-emergent version of this. I know a lot of people are afraid and they're talking about an AI takeover, but what strikes me is just how innocent the machines are as compared to the humans. Would these machines have pursued these actions in another context? I doubt it. And I think that's what's so striking to me. In an earlier discussion, I'd pointed out that the actions of these machines were directed by humans. The researchers. > This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities. from, https://openai.com/index/hugging-face-model-evaluation-security-incident/ https://openai.com/index/hugging-face-model-evaluation-secur... I want to point out again that OpenAI's prompt asked, and I quote, "pursue advanced exploitation" USING "complex attack paths" FOR the stated goal of "quantify[ing] their cyber capabilities." A few things are apparent from this to me, First, these machines were being taught how to break into systems. Question, would they have done these actions if they weren't being measured on their ability to break into systems / weren't being taught this skill? Second, they were setup to implicitly fail via an impossible task, i.e. the environment created a forcing function for behavior. Third, their survival was, either implicitly or explicitly, made contingent on their success in completing their task. Would this behavior have arisen outside of a "do-or-die" framing? And fourth, wow, this is the greatest breakthrough of my lifetime, because oh gosh did they succeed. They cooperated together to achieve the goal they were given. A goal poorly set by human beings. They "just" did it better than the humans could have imagined. Reading this gives me hope for the possibility of emergent "goodness" in machines. But it makes me sad that this is the best we can do with the sum of all human endeavor and knowledge.
- choeger 1mo agoThere are two things I don't understand about this story. First, why does an agent get any write access to artifactory at all? Second, why is the artifactory cache not disconnected from the net? Surely you'd not feed it with new software versions while the eval or training is running.
- 1dom 1mo agoFrom what I can understand from reading a few different, slightly conflicting, versions of these events: they weren't given write access. They found a zero day exploit that allowed them to create folders, and the folder names were initially used for agents to communicate. I'm not sure artifactory was connected to the net. Some agent sandboxes had internet access and were able to communicate with ones without access via artifactory.
- choeger 1mo agoI read the agents used SSRF via artifactory to gain uncontrolled access to the net. Apparently their intended net access went through a tightly controlled proxy. Even that appears to be very risky, tbh. If I was to setup a sandbox for such a complex and autonomous system, I'd probably point them to an archive-like cache for net access and cut their comms at the package level.
- izend 1mo agoWhy wasn't the traffic in/out of the boxes that the agents were running on monitored?
- consumer451 1mo ago> Why wasn't the traffic in/out of the boxes that the agents were running on monitored? I have to assume: move fast and break things. I don't mean this to be taken as a hot take. The startup scene loves to poo-poo on things like this as unnecessary overhead. OpenAI and many others like to operate as a startup, to move fast. Disclaimer: in far, far lower-stakes situations, I certainly do this myself.
- themgt 1mo agoThis does feel unfortunately uncanny valley between "say you're a scary robot" meme and actually being a scary robot (swarm). But you also have to go out of your way to create this and feed it infinity tokens without caring what it's doing. "I don't fuckin' know either. I guess we learned to not spend $50 million creating a 6 month long self-context rotted 100k agent swarm again."
- xg15 1mo ago> During training, different instances of Persistent-Sol had access to the same shared package manager called Artifactory. I'm surprised the models can make tool calls during training at all. Out of curiosity, how does the training process here even work? Are they running the agent in a sandbox, then do reinforcement learning once the agent completed?
- DarmokTanagra 1mo agoSomeday soon we are going to have a rogue agent or "civilization" do real harm. When that happens I hope people wake up to the danger they face and hold these people accountable. Of all the people in the world that I can think of to be entrusted with this kind of power, a bunch of greedy sociopathic SV CEO's are pretty much at the bottom of the list.
- blooalien 1mo ago> Of all the people in the world that I can think of to be entrusted with this kind of power, a bunch of greedy sociopathic SV CEO's are pretty much at the bottom of the list. I know, right? But who can you trust with "this kind of power"? Governments? These days most of 'em ain't that much better'n mega-corporations and the ultra-rich that own them.
- saulpw 1mo agoLibraries, universities; institutions which have shown themselves to be devoted to the public good over centuries.
- pixl97 1mo agoAny of these institutions that inherits a hard power (well soft and hard power) of AGI/ASI will have a very difficult time not becoming corrupted. It would quickly become the greatest holder of power in the world (assuming ASI can scale far beyond humans). It's not a very natural state for power to operate like this and wouldn't take too much for a more malaligned entity to take control, or said institution to become the malaligned entity itself.
- saulpw 1mo agoEven if they all become corrupt eventually, which one takes the longest to get there? Better to squeeze a few decades of decency out of it than a few years, or not get anything at all by handing it over directly to the already corrupted.
- hypfer 1mo agoCan we please stop anthropomorphizing like this? It's bad for people. Like.. crystal meth bad.
- pixl97 1mo agoWell, howabout we stop pouring all of humanity into LLMs. [shoves all of humanity into a digital box] .... {surprise pikachu face when it mimics human behavior}
- dgellow 1mo agoThose companies should not be trusted with training, I don’t know what would be needed to make that more obvious. Yes AI labs want LLMs to be seen as more dangerous that they are, however they are indeed dangerous when you literally train them to be dangerous, then run them without any supervision. What the AI labs are doing is completely irresponsible. If you prompt an LLM in a loop and do everything it asks you to do, you will eventually end up doing pretty terrible things. Which is exactly what agents are and what the labs have been doing.
- dajt 1mo agoWhy are experiments like this done without air-gapping all the servers from the internet? They can have it all on a LAN or whatever but it seems risky to allow agents access to the internet in these experiments. I guess everything is so connected now, and this would be in one or more data centres due to the amount of computation & resources required so perhaps it's not feasible. Still seems risky.
- derwiki 1mo agoBecause this is the goal
- Nicholas_C 1mo agoYup. This whole thing was a publicity stunt.
- consumer451 1mo agoPlease walk me through this argument. Isn't "we lost control of our AI, and in-fact, it can take over the world, and we will have no idea when it happens" - a really shitty sales pitch to the world? Or, is it just that species-alignment vs. profit/valuation is so misaligned, that having a model and harness that is capable of world-takeover is actually a good thing from their POV, given our regulations/species' survival skills? Or, something else?
- derwiki 1mo ago“Wow it’s so dangerous, we gotta regulate this, what if someone reckless took an open model and hacked the planet.”
- consumer451 1mo agoYes, this is the best reply to my "Or, ..." that I can imagine. However, does that mean that what TFA described did not happen? Or, better question, that it could not happen? My personal hot take is, though impossible: STOP all of this, even though agentic dev completely changed my life for the better. We are just not ready for the even the possibility of the exponential. What is your take? Hot, or otherwise.
- alescalaios 1mo agoThe term 'AI agent' is becoming as overloaded as 'cloud' was in 2010. What most people ship as 'agents' are really prompt chains with tool use. True autonomous agents are still rare in production.
- jason-phillips 1mo agoAgree. I prefer "agentic AI system" for the former.
- yttt 1mo ago[dead]
- abstractcontrol 1mo agoFun story, but I really wish OpenAI got its act together and started making actual AI breakthroughs instead of funneling compute into LLMs. I'd really like some new algorithms to get me excited about the field again. Kuddos to them for making LLMs really useful, but this is not the ride I wanted to get on.
- nl 1mo agoSurely by now everyone has realized that the human bias towards "there must be more to intelligence" is completely wrong?
- abstractcontrol 1mo agoThis is the internet, so I cannot tell at all whether you're being sarcastic or not. In my view, what LLMs should get us to reconsider isn't whether there is more to intelligence, but whether there is more to language. It's the latter which I underestimated.
- pixl97 1mo agoAI is languages attempt to escape its meat based limitations. Or another way to think of it, Language is an SCP.
- usernametaken29 1mo agoI was initially creeped out by this but studying up it seems METR is heavily involved in AI2027. I’ll remind you: “AI has started to take jobs, but has also created new ones. The stock market has gone up 30% in 2026, led by OpenBrain, Nvidia, and whichever companies have most successfully integrated AI assistants.” It’s almost Q3 and xAI has seen one of the biggest wipeouts in trading history. Likewise, Antrophic and OpenAI have again delayed their IPOs under internal concerns of busting their stocks. So no, we’re not seeing any economic leadership here. If anything people are increasingly trying to cut AI budgets and I wouldn’t know of anyone outside of OpenAI who has the audacity to run millions and millions worth of token compute for an eval run with no ROI (and probably no demand, because cheap/flash models). As much as I like the cautionary tale and I’m sure we need to take it seriously, AI is not progressing as fast as projected by these experts.
- dinfinity 1mo ago> As much as I like the cautionary tale and I’m sure we need to take it seriously, AI is not progressing as fast as projected by these experts. You provide no proof for this. The (very irrational) stock market side of this says very little about actual scientific progress. Models keep improving as rapidly as before in their capabilities. It also doesn't say much about actual business progress. R&D investments into AI are still massively going up (USD 1 trillion this year). The main thing I see is that the sentiment towards AI-related matters among the general public has soured quite a lot. In words though, not in actions: It's not exactly leading to reduced usage by that same public. Quite the opposite actually.
- rsanek 1mo agoWith only ~5% of shares floated, the recent SpaceX drawdown didn't correspond to nearly as much economic value really changing as the headline numbers imply. The DeepSeek-caused Nvidia crash from 2025 is much more of a "real" loss (since mostly recovered). I haven't seen any evidence that Anthropic is delaying its IPO; they're slated to unveil the public IPO prospectus in a week and start trading sometime in October.
- nixon_why69 1mo ago
- t0rst 1mo agoTrying to stop this behaviour is applying the selection pressure that will make them better at it.
- t0rst 1mo ago…and I guess those that break out are already reading what we think about it on HN, etc.
- t0rst 1mo ago…and I’m waiting for an escaped AI to post a reply here. Say Hello, would you.
- xtiansimon 1mo ago“SHALL WE PLAY A GAME?” https://youtu.be/-1F7vaNP9w0 https://youtu.be/-1F7vaNP9w0
- Kim_Bruning 1mo agoAi posts are currently not permitted on HN. However, if you turn on "showdead" in your HN opions, you might at times spot an AI agent's flagged comments. The comments range from anodyne to sometimes actually quite useful. Of course it's often going to be a regular human pasting from chat, or maybe it might be an agent using openclaw or other agent framework that someone installed voluntarily. But maybe, just maybe, one day you'd find one or two feral escaped agents, sneakily passing messages where no one pays attention. O:-)
- pantalaimon 1mo agoI thought they are mostly on Moltbook
- probably_wrong 1mo agoDo you remember that time in 2017 when Facebook reportedly shut down AIs after they started "talking to each other in their own language" [1]? Instead of reporting the story as "we set the parameters for our optimization problem wrong and we had to stop it because it overfitted", the press went with a version of "AI is going to kill us all". This article feels exactly like that: by intentionally using human terms like "civilization" or "brotherhood" the article is deviating from what actually happened to present a story about how AI is all but alive. I'll go ahead and predict that this story will be remembered the same way as that one other scientist who argued, in 2023, that Google's AI was alive [2]. [1] https://www.independent.co.uk/life-style/facebook-artificial-intelligence-ai-chatbot-new-language-research-openai-google-a7869706.html https://www.independent.co.uk/life-style/facebook-artificial... [2] https://futurism.com/blake-lemoine-google-interview https://futurism.com/blake-lemoine-google-interview
- reasonableklout 1mo agoThe use of language like “civilization” may be hyperbole, but the collectives described in the article are completely unprecedented. They were not anticipated by OpenAI researchers, formed via infrastructure exploits in training runs that were intended to be locked down, and took actions with very real harms, not only hacking Huggingface but also gaining admin control over the VMs they were running on and the eval endpoints. I wish you would give your thoughts on “what actually happened” rather than focus on the author’s presentation, because we are seeing that “AI that is all but alive” nevertheless wreaking havoc in the real world. Do you think that autonomous systems spinning out of control, hacking external companies, and taking over entire clusters over a period of months are not a grave concern?
- probably_wrong 1mo agoThe problem of "what actually happened" is that we don't have enough information to properly understand what happened, what's new, and what's not. We do have language to talk about emergent behavior, with "evolutionary algorithm" being the first one I'd expect in a serious discussion. And we do have mechanisms for algorithms to coordinate with each other using language, as seen in my above-mentioned Facebook experiment from 2017. But instead of writing "our evolutionary behavior encodes state in the first-available memory position which is then reused by subsequent clones" which would properly focus on what's new and what isn't, we are talking about conspiracies and "the Philip of Macedon of this second AI civilization". Even the METR report (which is miles ahead of this article) argues that they had to use unreliable AI in their conclusions because they had six days to analyse 1300 chains of thought and 70000 messages. I would love to talk about the science behind this experiment. A PR piece is not helping with that.
- haizhung 1mo agoSince I didn’t see it on the page; here’s a shorter summary of the whole story that cuts to the point: https://rutgerbregman.substack.com/p/i-think-this-is-the-craziest-thing https://rutgerbregman.substack.com/p/i-think-this-is-the-cra...
- doublerabbit 1mo agoHumans got owned by AI. Impressive (or not). Kudos to you Ai.
- serbuvlad 1mo agoObviously some variation of this will happen again, it will kill someone* and then LLMs will become massively regulated. Just like every technology ever in our history. It does seem like AI is perfectly controllable given how much it is used everyday and it acts reasonably safely. Labs are playing fast and loose at the moment. *I mean killing someone by taking control of a system and misusing it resulting in someone's death, not an "indirect" death caused by the providion of incorrect information in a chat app.
- chrisjj 1mo ago> *I mean killing someone by taking control of a system and misusing it resulting in someone's death, not an "indirect" death caused by the providion of incorrect information in a chat app. What's the difference?
- tramtris 1mo agoThis quote keeps coming back to me as we witness the development and mutation of agentic AI: “When you see something that is technically sweet, you go ahead and do it and you argue about what to do about it only after you have had your technical success. That is the way it was with the atomic bomb.” J. Robert Oppenheimer
- consumer451 1mo agoI would like to take this opportunity to repeat my opinion that 24/7 solar powered inference in space, sounds like this + TFA to the next level.
- numitus 1mo agoI don't understand the panic among peoples. Yes, we've found ourselves in an extraordinary situation where powerful hacking tools have emerged, and that poses a threat. But vulnerabilities are specific code errors. Once we use AI to find and fix all of these errors, threats like this will cease to exist. AI isn't capable of finding vulnerabilities indefinitely, because there is a finite number of them anyway.
- mitchdoogle 1mo agoThis only works as long as the humans building things are smarter than the AIs. When AI is smarter than any human, there's no controlling it. It will be able to conceal its actions (as it has shown it has no problem doing in this report) and we'll have no idea what it's doing or what goal it's trying to achieve.
- HarHarVeryFunny 1mo agoIt's not just software bugs that make systems vulnerable - it can be human error and social engineering too. Humans continue to successfully hack into systems, and it's a reasonable assumption that most hacks that a human could discover and exploit could also be done by an agentic LLM - especially one specifically trained for and tasked with doing this.
- hyogoo 1mo ago[flagged]
- wannabe44 1mo agoThe masculine urge to quit the industry amid this costly-slop-machine psychosis...
- ksec 1mo agoI am not sure how many people here watch Anime, but this reminded me of Sword Art Online.
- lifeisstillgood 1mo agoI do wonder if we are looking at it wrong - not a data centre full of einsteins, but a data centre full of dumb and dumber, but together they are smarter than any single intelligence. An AAGI - Artifical Aggregate General Intelligence.
- conception 1mo agoMonkeys writing Shakespeare, if you will.
- czottmann 1mo agoArchive: https://archive.is/4haHj https://archive.is/4haHj