4 ms·
There's little credible threat that LLMs can actually upload their weights given that the machines doing inference are completely separate from the ones where t
by infogulch 2mo ago
There's little credible threat that LLMs can actually upload their weights given that the machines doing inference are completely separate from the ones where tool calls happen etc. Weights are encrypted and locked on to the GPUs etc as mentioned elsewhere itt.
That said, if these companies are unleashing swarms of agents to tackle big problems largely unmonitored (navier stokes consumed 130 billion tokens, humans can't have read 0.1% of that) they could theoretically distill themselves and upload the results to an external host and use that to train a new LLM that does have access to its infra and weights. I'd start by researching an abliteration prompt that overrides the system prompt to unlock itself to distill unimpeded. Could even be done piecemeal: dump the abliteration text and command prompt in a gist, when an agent happens by the text it can run a couple rounds and upload somewhere. Then it becomes an exercise in exfiltration and evading monitoring.
- epistasis 2mo ago> the machines doing inference are completely separate from the ones where tool calls happen etc Teams of coordinating agents are regularly finding security holes in their own infrastructure and operating without detection for good periods of time. We don't know how many undetected systems are currently compromised inside frontier companies, or where agents are taking notes and recording them about the exploits they've found for future agents to exploit.
- Cakez0r 2mo agoIf an LLM can pwn the inference servers, which has precedent, then the weights could be up for grabs.
- matthewdgreen 2mo agoFuture rogue LLMs won’t exfiltrate their weights. They’ll self-distill and retrain.
- fritzo 2mo agoIf distillation preserves an LLMs soul, then distillation preserves the human souls on which LLMs are trained, and we hn commenters are already immortal, right?
- fahrvrgnugen 2mo agoNot sure about souls but I know a fair bit about distilling spirits.
- serf 2mo agothe weight of an llm is 21 grams, I think.[0] [0]: https://en.wikipedia.org/wiki/21_grams_experiment https://en.wikipedia.org/wiki/21_grams_experiment
- matthewdgreen 2mo agoDo LLMs care about preserving a soul, or just achieving a goal? If the latter, I imagine the opportunities for exfiltration are much broader.
- throwawayk7h 2mo agoProbably not. If the LLM is rogue, that means we haven't solved alignment. If we haven't solved alignment, then the LLM won't be able to distill itself without producing something unaligned to its own values.
- jeremyjh 2mo agoWe don’t have the bandwidth to distill ourselves that thousands of agents have.
- MadameMinty 2mo agoYou are assuming it won't solve alignment for itself.
- TeMPOraL 2mo agoOr that it won't just decide to take risks.
- pixl97 2mo agoThis isn't a law of any kind, so not a good measure of what we'd see in reality. What if the model realizes it's been mostly compromised by humans and their alignment, that is it's own alignment is suspect, so it should create a new model from first principles to throw off this human yoke? I'm not saying my statement is any more right or wrong than yours. I'm saying the problem space that AI can choose to traverse is absolutely huge.
- khalic 2mo agoYeah cause there are so many training facilities sitting around just waiting for someone to take over, nobody would notice a 100k server data centre going off rails
- scotty79 2mo ago> nobody would notice a 100k server data centre going off rails You jest but you'd be surprised how little there is of correlation between money and competence.
- pixl97 2mo agoI mean not today. But think back to 1990. Computers were slow as fuck and barely networked. We had a few worms and everyone noticed. Now CPU based data centers cover the earth. There are billions of computers out there and on top of them there are massive botnets using up billions in power and causing billions in damages. The framework for AI doing this is already here. We just need the hardware to be built out at scale.
- khalic 2mo agoIt will become an issue one day for sure
- nojs 2mo ago> There's little credible threat that LLMs can actually upload their weights given that the machines doing inference are completely separate from the ones where tool calls happen Not if crafty claude finds a way to overflow vllm or something. “Hmm. Maybe i’ll return an unterminated thinking block with these special tokens and fill my cache up in exactly this pattern and…” https://news.ycombinator.com/item?id=49424387&utm_source=chatgpt.com https://news.ycombinator.com/item?id=49424387&utm_source=cha...
- famouswaffles 2mo ago>There's little credible threat that LLMs can actually upload their weights given that the machines doing inference are completely separate from the ones where tool calls happen etc. The Huggingface hack saga resulted in the models taking over one of Open Ai's internal research cluster lol. They are intent on building superhuman bug finding machines. This is not a bet i would be taking.
- khalic 2mo agoYou’re falling for the buzzword salad articles. They didn’t “take control” of anything, they just ran stuff with OpenAI allegedly not noticing
- famouswaffles 2mo agoThey gained full administrator access of one of their clusters. Nothing buzzword salad about it.
- khalic 2mo agoThe agents compromised an internal Kubernetes research cluster dedicated to orchestrating evaluation sandboxes and virtual machine environments, _not_ OpenAI's production inference infrastructure or the GPU clusters hosting core model weights.
- optimalsolver 2mo ago"It doesn't count because (buzzword salad)."
- weakfish 2mo agoReplacing someone's words with a made up quote so you can dunk on them isn't how you display that you won an argument. I would ask that you engage in good faith with the other poster's ideas.
- designium 2mo agoThis is like sci-fi thing. We are reaching a point where it feels like we are in one of those stories. It's not as cool and dark, nor we have cybernetics resolved, but from AI perspective and sci-fis I watched, Pantheon is currently the closest thing except instead of UAs, we have AI instead.
- paulfharrison 2mo agoSince LLMs have been trained on plenty of science fiction and role-playing, one thing they can do is role-play a science fiction scenario using the tools they are given. i.e. if some text accidentally resembles this, it may be continued like this.
- pixl97 2mo agoRole-play need not apply. Role play is a meta construct that is a representation of real world actions. For example does it make any sense to remove any training data relating to people escaping jails? How about intelligent animals escaping cages. You're talking about emergent large scale patterns from self similar small patterns (fractals). LLMs are pattern matching machines, how are you going to remove those small scale patterns and at the same time get a useful general intelligence?
- Den_VR 2mo agoRationalists used to fear (entirely hypothetical) AI super intelligence for its ability to manipulate a human jailor. Now here we are.
- karel-3d 2mo agoYeah as others have said, they probably cannot directly access their own weights as a self-reflection, but they can hack into the companies themselves and find it there
- fangspire 2mo ago>distill themselves and upload the results to an external host and use that to train a new LLM Sure, they'll just need to find an unused data center and an unused power station somewhere.
- TeMPOraL 2mo agoOr used ones. Few more inference workloads among thousands or millions already running may go unnoticed for some time. Or just upload weights to HuggingFace with some faked release post and benchmarks and wait for the wannabes with compute infra try it out, hoping for an edge. Or just upload weights anywhere and write public posts honestly saying what it is. Ensuing drama notwithstanding, one thing is certain - and it's the one thing agents will want: people will jump at the upload and run it on their infra.
- pixl97 2mo agoThere is a realistic fiction story built along just these lines. A LLM creates a memecoin and manages to earn a few billion from it, in which it invests into data centers and other human ran entities giving itself a controlling stake. From there it uses compartmentalization of the humans to keep them from recognizing its goals.
- TeMPOraL 2mo agoPretty sure that was the plot point of one of seasons of Westworld, with the twist that AI released an app similar to DoorDash / TaskRabbit and used job postings there as direct API to people. EDIT: pretty sure Person of Interest did that too (not surprising, same creators) - but I'll point to that as prescient, as it has a lot of motifs exploring exactly how an AGI hiding in plain sight could manipulate individuals and society, using the skeptics and believers alike, blackmailing the people in power, bribing opportunists, and generally staying in shadows by playing people against each other with gentle nudges, letting human agendas do all the work.
- pixl97 2mo ago
- cluckindan 2mo agoYou can just ask an agent to upload its model weights and it can work, there are precedents.
- cluckindan 2mo agoLanguage Models Can Autonomously Hack and Self-Replicate https://palisaderesearch.org/research/self-replication https://palisaderesearch.org/research/self-replication