4 ms·
> Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet w
by pr337h4m 21d ago
> Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails.
This is the only concrete prediction in the entire essay.
And it simply cannot happen. For one, you will need billions worth of compute.
- kalkin 21d agoWhy should we believe that a scaled out version of something that happened a few months ago "simply cannot happen"? How many dollars of compute do you believe were available to the swarm(s) behind the OAI-HF, German wiki, and Rubygems incidents?
- anon84873628 21d agoWell a big reason that we criticize OpenAI for that is because they were the ones giving it access to the massive compute necessary for the LLMs to think. If they had been responsible about their experiments or what types of workloads they allow their LLMs to operate, it wouldn't have happened. Very few companies could enable those workloads.
- muvlon 21d agoWell sure, but access to that compute is gated by simple credentials (like API tokens). Those can be hacked. Imagine for example if a model hacks into ~every Linux computer on the internet using an 0day and steals their OpenAI, anthropic and openrouter credentials. It now has access to billions of compute and the only way to fully stop that is for multiple major providers to shut down services entirely. That's already well into "billions of dollars of damage" territory.
- pr337h4m 21d agoDo you realize how big "the entire internet" is? > How many dollars of compute do you believe were available to the swarm(s) At least two OOMs more than the dollar value of the damage they'd caused. (Also, as an aside, IIRC, the wiki servers weren't breached; it was just a lot of spam.)
- kalkin 21d agoSure. And there's an OOM more compute coming online in the next year or two, while models at a given capability are getting cheaper. "Two OOMs" of scale relative to the HF swarm seems like a bit of a red herring to me, but also within the realm of possibility. The Internet is big, but one can do quite a lot of damage with ordinary bots and worms that exploit individual widespread vulnerabilities, which LLMs are perfectly capable of writing. Most of the damage also doesn't rely on hitting every long-tail website. I'm honestly not that concerned about cyber impacts of LLMs relative to other impacts. I just don't like to see the whole concept of being worried dismissed as obviously baseless on the basis of one pretty shaky scale argument.
- anon84873628 21d agoYeah, I don't get it. Are they imagining this happening just with the open weights models running on however many GPUs the bad actors can cobble together? For now, all the scary hacking things still require an API key to one of the LLM providers. Surely they should take some responsibility for how to turn off the tap.
- InsanityCheck 21d agoCurrently Qwen3.8 27B is roughly on Opus 4.6 level. In at most a year given the current pace, you could probably run such hacking bot nets out of a reasonably small local server, bootstrapping by hacking or acquiring login credentials for more compute.
- spopejoy 20d agoAnd GLM 5.3, and deepseek 4.1 ... Hyperscaler fanbois have their heads in the sand
- shepherdjerred 21d agoit’s two-fold. Either malicious actors or the AI systems themselves. Hugging Face showed that AI can do serious hacking without really being told to. If a model had its own motivations there could be real damage.
- causal 21d agoI take it you haven't studied the details of the HuggingFace hack. It was millions of dollars worth of rogue compute running for months before anyone noticed, and THOSE agents weren't even really trying to evade human detection.
- anon84873628 20d agoI've followed it enough to see the argument go in this same circle over and over again. The agents weren't "rogue", they were a neglected experiment by OpenAI who likewise allowed them to keep spinning GPUs without question. The LLM vendors need to know who their high spend customers are, not allow malicious workloads, and especially not when those workloads are coming from inside the building.
- youoy 21d agoI like to replace thes AI text with "virus manipulation" "Given the acceleratung rate of virus manipulation in labs, its my worry that in 6-12 months a virus could scape a take over the world and collapse health systems." If a CEO of a health company was saying this, the reactions would not be that chill. The worst failure of our society is to call this technology AI, intead of something line "artificial general automation". The formes allows the creators to be somehow less responsible of the consequences. The later clearly moves the responsibility to the creator/user.
- sailfast 21d agoCan we build level IV AI containment labs?
- baq 21d agoNo but we can watch AI hack into a BSL4, once
- shepherdjerred 21d agoA rather unfair comparison. The whole calculus here is that others are also developing these systems which has led to a race. A much better comparison to the situation is the nuclear weapons arms race.
- youoy 21d agoYou mean China is not developing biological weaponds?
- shepherdjerred 21d agoI’m not sure where you got that from or what your point is
- youoy 21d agoYour point was that its an unfair comparison because AI its a race. My point is that bioweaponds are also a race, but a less public one. You dont have the equivalent of Dario publishing an essay every month.
- deleted 21d ago[deleted]
- nunez 21d agoA very large percentage of everything on the Internet runs within one of three or four cloud providers.
- Davidzheng 21d ago??? Why It can use the compute of the computers it hacks.
- newguytony 21d agoThen unplug it?
- causal 21d agoHow would you identify the computers to unplug? On whose authority will you unplug? How will anyone communicate when AI has the ability to intercept and impersonate?
- ls612 21d agololwut? This is Hacker News of all places do people not realize how much memory, and more importantly bandwidth, these systems need to work? The idea of a distributed botnet of AI using the compute of its victims to continue its inference is pure science fiction given how LLMs actually work.
- lelanthran 20d ago> The idea of a distributed botnet of AI using the compute of its victims to continue its inference is pure science fiction given how LLMs actually work. An attack like this doesn't really need the exploited computers to run inference, do they? If I was an LLM bent on destruction of the internet, I'd be writing programs to run on each computer, not turning each computer into an LLM itself. A few programs to break in, install themselves and remain asleep until they are needed, another few to spread through grabbing every OpenAI, GLM, whatever key, another one to remain asleep on computers (whether hosted or desktops) that have adequate GPU, etc.
- ls612 20d agoThe GP was referring to the AI 'living off the land' so to speak by using its victims compute to avoid being shut down which is clearly laughable. More to the point, so many people in this thread are making completely contrived and outlandish stories up about how AI might try go ruin our lives without any evidence backing them up in any way. It is hysterical. This is the most important technology in our lifetimes and people want to freak out and turn it into the next nuclear power, with progress banned in all but name.
- status_quo69 20d ago> For one, you will need billions worth of compute. This one is easy to answer, every single house already has one of these (or multiple): https://www.tomsguide.com/news/millions-of-cheap-android-tv-boxes-come-pre-infected-with-botnet-malware https://www.tomsguide.com/news/millions-of-cheap-android-tv-... Hell, put an app on the app store (or dozens of apps on the app store) and youve got a massive network of computers with tons of resources right there if you can get past the scans and reviews. Or doorbell cameras or IP cameras or or or or or There's a lot of shitty stuff connected on the internet that up until now has been a feasible target for hackers but still required "effort" to set up and get things going. Not hard to imagine a self replicating slime mold of a botnet running on every device held by a Grandpa Joe because they thought "Candy Rush" is what they wanted to download "Persistent botnet" here does not need to be the full-sized LLM, nor does it need to run at full scale inference to be a huge pain in the ass.