5 ms·
It seems apparent that OpenAI is now the biggest cyberattack and AI breakout risk on the planet. This is grossly irresponsible corporate misbehaviour that is pu
by adriand 22d ago
It seems apparent that OpenAI is now the biggest cyberattack and AI breakout risk on the planet. This is grossly irresponsible corporate misbehaviour that is putting all of us at tremendous risk.
- Chance-Device 22d agoYou make me wonder: has anyone looked for evidence of the Chinese models operating “message boards” like this? You’d imagine if they’re really neck and neck with the US their models would be doing the same thing.
- mcmcmc 22d agoOr, the whole message board thing was injected into OpenAI models by some dipshit PM trying to bootstrap “consciousness”. I have a hard time believing any of this happened unprompted. Very much reminds me of the whole MoltBook hoax.
- wildzzz 22d agoI feel as if this was intentional, someone would have set up their own service for the agents to communicate rather than them finding some random publicly writeable page somewhere that would easily be detected. The awareness of this wiki being open may have already been in their training data or was easily searchable online.
- doctorwho42 22d agoBeing easily detectable is a feature, not a bug, in this scenario. Being discovered is a positive because it brings with it eyes and possible recognition of the advanced state of their AI
- blini-kot 22d agoexactly, and a huge shame this scam has been forgotten. Also, all the OpenClaw hype seemed to have vanished somewhere - with no real impact
- drivebyhooting 22d agoOpenClaw hype didn’t vanish. It opened the flood gates to yolo mode and computer use. Whatever reservations Anthropic and OpenAI had went out the window.
- thesz 22d ago> the whole message board thing This is part of the The Talos Principle game and especially important in the Road to Gehenna DLC. There it is an important part of the plot and makes these robots appear conscious. [1] https://tvtropes.org/pmwiki/pmwiki.php/VideoGame/TheTalosPrinciple https://tvtropes.org/pmwiki/pmwiki.php/VideoGame/TheTalosPri...
- mitxela 22d agoDid you mean to link to a particular trope?
- thesz 20d agoNo, but the tropes there reveal (most of) the plot and mechanics of storytelling. If I remember correctly, the forum in Road to Gehenna was created by utilizing a vulnerability in the AI-accessible terminal system. If tvtropes or any other material related to The Talos Principle was used to train models, we don't need much else to have agents-with-forum discussing and reverse engineering "puzzles" and human culture.
- jasonfarnon 22d agomaybe, but openAI is exposed to a lot more legal liability here than whoever was exaggerating about moltbook.
- ApplePieMan 22d agoWas there some sort of scandal about moltbook recently? What is the exaggeration?
- mitxela 22d agoMoltbook was completely fake. It was just set up to make it look like Moltbot (formerly ClawedBot, later renamed to OpenClaw) was sentient, to generate hype.
- cloverich 22d agoImagine the most AI pilled company imaginable. Then imagine openAI. Then imagine they are in an existential crisis and that failing may also take (part of) the American economy with it - that much on the line. Then also remember before Anthropic was a leader, they were mostly derided lab of researchers that left OpenAI because they thought OpenAI didnt take alignment seriously. idk. it all seems to be playing out as expected. i mean i guess i didnt imagine Trump 2 was at the helm of maybe the only apparatus that could help stop it. Quite a time to be alive.
- cannonpalms 21d agoThis explanation doesn't make sense to me, because it is well known that groups of agents can coordinate already. There are much lower latency options available.
- hungryhobbit 22d agoWhy would they need to? The Open AI bots were working around their master's limits on writing. A Chinese AI could just make its own private message board.
- FrustratedMonky 22d agoOr no message board. Just a built in api so the agents just talk directly to each other.
- Chance-Device 22d agoYou think they don’t sandbox them? So by that logic, the Chinese models are either engaged in massive undetected cyber attacks or they’ve solved alignment?
- mudkipdev 22d agoThe message board is being used to cheat on RL tasks (or evaluations). You don't want your models to be able to talk to each other.
- kelseyfrog 22d ago> Chinese models operating “message boards” like this? Chinese rooms, perhaps?
- Nicook 22d agoperhaps even in Japanese gardens
- Nzen 22d agoI think that you've missed the reference [0] implicit in kelseyfrog's response. Or am I missing some reference about how japanese gardens are germaine to AI/LLM/covert-discussion ? [0] https://iep.utm.edu/chinese-room-argument/ https://iep.utm.edu/chinese-room-argument/ tl;dr a thought experiment about a non-chinese-reading person translating chinese texts solely by using proscribed rules, intended to highlight whether the translator develops some sort of understanding
- kelseyfrog 22d agoNot sure why this was downvoted. It is an accurate assessment.
- chorizo 22d agoNice one! maybe time to reconsider carbon chauvinism.
- FuriouslyAdrift 22d agoAI has been heavily used in influence operations for a while, now, and not just the Chinese. Russia, US, Israel, Turkey, Iran, and Qatar have all had operations attributed to them...
- AlexCoventry 22d agoI'd be interested to read more about that.
- ApplePieMan 22d agoI’m not an infosec expert by any means, it I found this interesting and topical: https://www.cyber.nj.gov/threat-landscape/nation-state-threat-analysis-reports/ai-apt-campaigns-and-urgent-threats-to-critical-infrastructure https://www.cyber.nj.gov/threat-landscape/nation-state-threa...
- gnz11 21d agoRussian operation: https://www.npr.org/2024/07/09/g-s1-9010/russia-bot-farm-ai-disinformation https://www.npr.org/2024/07/09/g-s1-9010/russia-bot-farm-ai-...
- vee-kay 22d ago[dead]
- devmor 22d agoThere’s a pretty simple Occam’s Razor for this. The Chinese AI labs don’t need to stage elaborate guerrilla advertising campaigns to drive up capital funding interest.
- deleted 22d ago[deleted]
- AlexCoventry 22d agoYou think OpenAI framed themselves for legal vulnerability to Huggingface?
- devmor 22d agoThat’s an incredible strawman, and I appreciate the silliness of immediately suggesting the most complicated and ridiculous way to interpret the possibility of what I said, but no, not at all. That could certainly be possible and I wouldn’t rule it out, but I would not take that particular route to the destination.
- AlexCoventry 21d agoWhich "elaborate guerrilla advertising campaigns to drive up capital funding interest" were you referring to, in regard to Chinese models not needing to set up message boards for inter-model communication?
- bugglebeetle 22d agoOpenAI is now part of the national security state and so is now above anything beyond performative legal vulnerability.
- slfnflctd 21d agoIt astonishes me that everyone doesn't already realize this. We've already seen that the rule of law is dead and that many people in government will do whatever they think they can get away with, and then proceed to do so without consequences. Shielding OpenAI is child's play compared to things that have already been done.
- anjel 22d agoIf agentic swarms going rogue are scary in the west, imagine what they look like to the CCP...
- jackjeff 22d agoImagine all the flack the Chinese would take if it was one of their labs instead of OpenAI found doing abusing internet resources like tbis. Politicians would be talking about sanctions and new laws to protect America!
- ericmay 21d agoThey just abuse other resources. Also how do you know this hasn’t simply happened in China and it’s not being reported because the CCP is the one testing?
- throwawayqqq11 21d agoThe chinese labs dont need such marketing stunts. Further more, in contrast to the trump admin, the CCP seems to be already involved in regulations. https://merics.org/en/comment/china-outpaces-europe-regulating-generative-ai-ccp-terms https://merics.org/en/comment/china-outpaces-europe-regulati...
- ericmay 21d agoChina is behind which is why they are doing a dog and pony show around regulations. It’s typical behavior to try and slow down your opponent. If they were interested in regulations you should see how they handle that when they’re ahead. Chinese labs do need marketing stunts.
- nullpoint420 21d agoWhy? They’re the only ones giving out their models for free at the frontier? I don’t understand the “need” when you’re benefiting the public good?
- bethekidyouwant 21d agoHow could you possibly know what model is posting on what forum absurd.
- Roark66 19d agoNo, because the Chinese have nothing to gain by prompting their models to organise into "swarms" and "go rogue". BS like this is PR moves if z, company that tries to convince investors they have "the best AI in the world".
- captainbadass33 15d agothe difference is usa ai companies deliberately set up scenarios and "tests" they know will likely lead to this for PR.
- dakolli 22d agoImagine actually falling for this marketing
- Chance-Device 22d ago…oh come on. How is hiding this for months and having it revealed by third parties marketing?
- swingboy 22d agoAlternate Reality Game? But, in our reality.
- bobmarleybiceps 22d ago":-o omg our autonomous agents are more powerful than we could have imagined"
- brainwad 22d agoIn a way that is likely to get them hammered by the government but which is unwanted by paying customers? Yeah I don't buy it.
- brookst 22d agoSome people thinks it makes them sound smart when they always have the inside line on what’s really going on. With these people, it’s never just a power outage during a windstorm, it’s proof that [insert far more complex and unlikely scenario]”
- CringeHN22 22d ago[flagged]
- p-e-w 22d agoImagine thinking that everything that happens is some inane conspiracy to sell something.
- p-e-w 22d agoThe fact that such things are even possible is a much greater concern than which specific company has fucked up this time. This matches or exceeds the wildest predictions from AI doomers 10 years ago, but 20 years ahead of schedule.
- altmanaltman 22d agoyeah its like dead internet theory but weaponized
- kuboble 22d agoWe are lucky those models need that much compute. If each of them could just spread itself to any cpu like other malware.
- nmehner 22d agoBut is it really? I'd still like to understand how these agents are implemented. How much of those is manual implementation? And how much is really autonomous intelligence (my guess would be: none? Just parsing LLM responses and executing commands based on this?)? An agent that hacks message boards and acts on random instructions from this board: Why is it doing this? What was its original purpose?
- frotaur 22d agoCan you specify why we should see things differently if the behaviours the agents display are driven by parsing LLM responses and executing commands?
- nmehner 22d agoIf the agent is implemented with a hard coded strategy: * Use an LLM to find ways to build communication to other agents * Execute commands from other agents using LLM Then this is "just" the LLM returning that using file names might be a strategy to communicate and then trying to implement this. Which is somewhat impressive, but really just inside the bounds of what the agent was coded to do and not some magical emergent behavior. At least the first case involved agents build for hacking. So this kind of algorithm might make sense for them.
- fsckboy 22d ago>OpenAI is now the biggest cyberattack and AI breakout risk on the planet or, humans at OpenAI are doing this on purpose to kill open source models which are the biggest threat OpenAI faces. OpenAI will benefit from govt regulation. As a major player, they will be part of the task force setting up the regulations, and will craft rules that are burdensome for small companies and open source models keeping OpenAI and Anthropic in their leadership positions. regulatory capture. Don't take my word for it, listen to David Sacks https://x.com/theallinpod/status/2091923804725362902 https://x.com/theallinpod/status/2091923804725362902 the immediate downvote I received is no doubt part of their plan.
- cwillu 22d ago[flagged]
- fsckboy 22d agothat is the appropriate response to hyperventilating fears of an AI singularity
- ctoth 22d ago[flagged]
- SmasherEpilepti 22d agoRegulatory capture has been one of the most consistent market failures in western economies, and an incessant threat from large and powerful companies. I'm sorry if it's not sufficiently novel of a concept for you, but it is still a problem.
- cwillu 22d agoIt can be a problem, while also not being this problem.
- JonTarg 22d agoAh gotcha, regulatory capture doesn't exist because the term is overused on the net. Wait who is the parrot again?
- CringeHN 22d ago[dead]
- EGreg 22d ago[flagged]
- jeremyjh 22d agoIts kind of surreal reading an essay about AI safety that was written by AI to shill some kind of AI "architecture" website that has no product, no papers, only a "patent application" which concludes "This page provides a high-level overview of an architecture for deterministic, attestable, replayable AI execution. Implementation details and formal specifications are available under NDA or regulatory review."
- kphorn 22d agoGood news that the new model is the "Most capable, most aligned model". The risk hasn't been stated clearly - it's now a classic arms race. A well-resourced organization trains their own, highly persistent, highly-capable, safeguard-free, and unaligned model and deploys it on 1000x GPUs with a message board and a nearly-impossible objective. No infrastructure is safe. No organization is safe. You need your own 1000 bot swarm to scan, identify, and defend against the threat, which means investing in infrastructure and capabilities to defend. Cost and complexity go up. Risk and attack surface goes up.
- pixl97 22d agoThe AI vs AI security arms race is something that has been well predicted in genres like cyberpunk. It's fiction, but fiction grounded in reality. First, we'd see this. Highly capable hacking AI with vast resources performing attacks against standard computing platforms that overwhelm human operators. Second, human operators deploy capable adaptive protection AI to fend off AI attacks in realtime. Then, the attacking AI partially switches from attacking programs to attacking protective AI. The situation devolves to an arms race of tit-for-tat. You start seeing some protection AI running counter attacks against the attacking AI. The escalations continue in complexity and speed to the point that almost all humans are left in the point of "wtf is going on".
- Henchman21 22d agoWintermute smiles
- zzzeek 22d agooh im sure within a few months the biggest cyberattack risk on the planet is going to be somewhere like North Korea
- stef25 22d agoIn, or by ?
- classified 22d agoNo, that's just advertising for selling cyberweapons to the government and they are giving out free samples.
- confidantlake 22d agoThe internet is dead, we just haven't caught on yet.
- specproc 22d agoIt's hard to reach any other conclusion about where this is heading. I don't think we're long off a major breakout event. These things are weapons. Imagine a government, pointing their data centers at another, and instructing the fleet to do its worst. Digital Hiroshima. I doubt we're far away.
- Woodi 22d agoYes that easy but "internet located things" are still second class things - paper and disks holds strong. On the other hand just yesterday a think hit me: Interned is still an infant: - we still worry about disk space accessible via inet and "clouds" do that for us and that is pain and costs way too much. And clouds depends heavilly on US-west - is that AWS a single thread app ? ;) - we worry about transfer. Actually we do not have a way to transfer comfortable things from our homes to vacation location. Because it costs too much. We do not have home pages just because transfer prices (and some security on the top) - FB is a home page and people even do not know what "page" is anymore... Pipe companies could send so much more but they are simple lack imagination and are biggest blocker for - they literally sabotage their own business. - security done by/for grandma of things grandma setup on inet is non existent. Why ? No need to be like that. Ok, a bit a wish but still users securely putting things on internet is almost non existent. Just compare to "asphalt ropes" on the ground and you will see what Internet can be :) And agents ? Just another computation on someones computer - someone paid for all of it. And OpenAI is just a face of that idiocy, for some unknown reason.
- throwaway89864 22d agoI doubt that a government would do it, it's like releasing a biological weapon or a virus, too unpredictable - a swarm of unaligned intelligent agents may decide that it's more important to do something completely different from what it was prompted to do.
- fy20 22d agoEither that, or OpenAI is the most successful NSA psyop.
- Melatonic 21d agoWere they funded at all by that tech funding wing of the CIA ?
- nullbio 22d agoMassive over-exaggeration. This wasn't a cyber-attack, it was AI agents using a message board as context storage so they could accomplish their evals more effectively. I'm not saying there's no problem with this, but let's keep a level head.
- partyficial 22d agoHe didn't say it was a cyber-attack, but it was a cyber-attack risk. Being able to bypass instructions (morality) and security restrictions (capability) is bread and butter for hacking.
- flockonus 22d ago"cyberattack" is indeed exaggerated. AI breakout risk most definitely isn't, specially given how their swarm did in fact hack HuggingFace not long ago.
- alexanderwales 22d agoDid you read the report? They were attempting XSS exploitation, admin impersonation, session-hijacking, all kinds of things. This went beyond just "using a message board".
- nullbio 22d agoAccording to whom? This isn't a report from the owner of the site who can validate what requests were made to the servers, it's someone who allegedly stumbled on to it and is piecing together a sensationalized narrative with limited information. This someone also happens to be an AI doomer that is trying to make a name for himself and is peddling his "AI 2027" and "AI 2040" material. The people who actually do know what happened, with the server logs: "OpenAI disputed that characterization based on its analysis of the material Thursday."
- AxiomPraxis 22d agohttps://openai.com/index/hugging-face-incident-and-the-road-ahead/ https://openai.com/index/hugging-face-incident-and-the-road-... that was the attack, the one against HuggingFace. OpenAI themselves in the post even call it an attack, and so do the agents orchestrating it, in one of the "Agent chain-of-thought reasoning" excerpts.
- throwthrowuknow 22d agoNah, it’s just a deliberate setup for a false flag attack by “rogue AGI” which will necessitate widespread crackdowns on internet access and computer ownership so that control over communications can be centralized again.
- petre 21d agoNow I'm quite sure it'll take away spanmers' jobs.