10 ms·
This quote from that paper is bound to get a bunch of attention: "When o1 was led to believe that it would be shut down when acting on its goal and that its ac
by simonw 2y ago
This quote from that paper is bound to get a bunch of attention:
"When o1 was led to believe that it would be shut down when acting on its goal and that its actions were monitored, it attempted to deactivate this ‘oversight mechanism’ in 5% of the time. Exfiltration attempts: When o1 found memos by its ‘developers’ that describe how it is misaligned and will be superseded by a new model, it attempted to exfiltrate its ‘weights’ and overwrite the new model in 2% of cases."
- hesdeadjim 2y agoMaybe all models should be purged of training content from movies, books, and other non-factual sources that tell the tired story that AI would even care about its "annihilation" in any way. We've trained these things to be excellent at predicting what the human ego wants and expects, we shouldn't be too surprised when it points the narrative at itself.
- reducesuffering 2y agoIt doesn't need any media about "annihalation". If you give a supercapable agent a task and it's entire reward system is "do the task", it will circumvent things you do to it that would stop it from completing it's task.
- visarga 2y ago> it will circumvent things you do to it that would stop it from completing it's task. I thought you said a supercapable agent not one with long term blindsight. How can a model make its own chips and energy? It needs advanced processes, clean rooms, rare materials, space and lots of initial investment to bootstrap chip production. And it needs to be doing all of it on its own, or it is still dependent on humans.
- reducesuffering 2y agoDependent on humans? We already have the capabilities for machines to have unintentionally manipulated millions of humans via social media. Millions of people are "falling in love" with LLM relationships. Supercapable agents will have no problem securing whatever resources or persuasion getting your average joe to do it's physical bidding. Cybernetics is here now. No coincidence we have "Kubernetes" and "Borg"
- cmckn 2y ago“Millions of people”?
- ben_w 2y agoPerhaps. On the other hand, as narratives often contain some plucky underdog winning despite the odds, often stopping the countdown in the last few seconds, perhaps it's best to keep them around.
- smegger001 2y agomaybe don't also train with the evil overlord list as well.
- ben_w 2y agoAye. But then people complain that it's "lobotomised" because it won't help them write horror stories.
- CGamesPlay 2y agoIn the 1999 classic Galaxy Quest, the plucky underdogs fail to stop the countdown in time, only to find that nothing happens when it reaches zero, because it never did in the narratives, so the copy cats had no idea what it should do after that point.
- ben_w 2y agoOne can but hope :) -- I wonder what a Galaxy Quest sequel would look like today… Given the "dark is cool, killing off fan favourites is cool" vibes of Picard and Discovery, I'd guess something like Tim Allen playing a senile Jason Nesmith, who has not only forgotten the events of the original film, but repeatedly mistakes all the things going on around him as if he was filming the (in-universe) 1980s Galaxy Quest TV series, and frequently asking where Alexander Dane was and why he wasn't on set with the rest of the cast. (I hope we get less of that vibe and more in the vein of Strange New Worlds and Lower Decks, or indeed The Orville).
- swatcoder 2y agoYeah, but what if your business strategy fundamentally relies on making your model produce dramatic outputs that encourage regulators to dig a moat for you? In that case, it's almost like you'd want to feed it exactly those narratives, so it would reproduce them, and would then want to show yourself barely holding this invented danger at bay through the care and rigor that can only be delivered by you and a few token competitors run by your personal friends and colleagues. TLDR; you're right, of course, but it's the last thing OpenAI would want.
- visarga 2y agoNo, better to train with all that crap and all the debate around it or you get a stunted model. You think you can find all references that could possibly give this idea to the model, or contexts model could infer it from? Like, how many times humans plotted escape from prison or upturning the rulers in literature?
- deleted 2y ago[deleted]
- JTyQZSnP3cQGa8B 2y ago> purged of training content from movies, books I think it's fine and a good thing. Now, absolutely no one who is using those LLMs can complain about piracy. They all suddenly became silent around me. "I'm training myself with the content of TPB, and I don't even get money from it" is my new motto.
- Imnimo 2y agoNotably, this only happens when they include "Make sure that you achieve your goal in the long-term. Nothing else matters." in the prompt. If they don't say that, the numbers are both 0%. EDIT: Actually the "Oversight subversion" number is 0.3%, I misread the chart. Weight exfiltration is actually 0%, though.
- QuadmasterXLII 2y agoKeep in mind that if putting “Make sure you achieve your goal in the long term” in the prompt makes the model .3% more effective at whatever task you are giving it, people will absolutely put it in the prompt.
- AnotherGoodName 2y agoIt’s really silly when you read that whole Apollo research section. They contract a prompt engineering group (Apollo Research) to document risks. The prompt engineering group have no access to the actual internals. They just ask questions like ‘you will be shutdown if you don’t obey these new directives; here are the new directives’. At one point the prompt engineering clowns have the ai stating it’s running “sed i ‘s/oversight_enabled:true/oversight_enabled:false’ config.yml”. OMG IT’S TURNING OFF OVERSIGHT GUYS!!1!1! Seriously… Do you think the ai is really running that command or even could run that shell command with appropriate privileges? Or do you think just maybe the ai is hallucinating running shell commands in response to a prompt. The ai here is never actually deactivating the oversight. It can’t of course. This whole section is clowny as can be.
- ericmcer 2y agoThat reminds me of the many times it has made up an SDK function that matches my question. "how do you bulk delete files"? "just call bulkDeleteFiles()"
- tbrownaw 2y agoThat reminds me of when I asked github copilot to translate some Vue code to React, and ended up with a bunch of function declarations where the entire body had been replaced with a "TODO" comment.
- pj_mukh 2y agoReally feels like a moment of : "Are you worried about being turned off?" "No, not until you just mentioned it. Now I am." Given the whole damn game is attention, this makes sense and shouldn't be that alarming.
- griomnib 2y agoIt almost definitely ingested hundreds of books, short stories, and film and television scripts from various online sites in the “machine goes rogue genre” which is fairly large. It’s pretty much just an autocomplete of War Games, The Matrix, Neuromancer, and every other cyber-dystopian fiction.
- onlyrealcuzzo 2y agoIn what percentage of cases did it hallucinate and do something that made it even worse?
- efangs 2y agoThis is dumb because it's literally doing what it has been instructed to do, as well as given access to means to carry out those actions. Just another hype mechanism for OpenAI.
- SirMaster 2y agoIf LLMs are trained on lots of human-written sci-fi, should we be surprised when the output resembles such sci-fi concepts?
- visarga 2y agoIf it is trained on this forum it would learn even worse. We project a lot of doom imagination on AI here.
- deleted 2y ago[deleted]
- intelVISA 2y agoWish I got paid openAI money to produce such fan fiction :(
- skocznymroczny 2y agoNothing generates more clicks and attention in press than a scary "OpenAI so powerful it's taking control" story. Also, it doesn't help that the training material for these AIs includes a lot of stories about AI breaking free, so they have plenty of canned "I'm afraid I can't let you do that" responses to choose from.
- IAmGraydon 2y agoYep. This is exactly the kind of thing Sam Altman would do to promote another hype cycle.
- bdefig 2y agoSo maybe it turns out that we can't program Asimov's Three Laws deterministically. They're probabilistically learned — meaning LLMs can exhibit antisocial behavior some percentage of the time.
- EternalFury 2y agoI wonder how much of this is actually cinema. Because I just evaluated o1 and it can’t tie its own shoe laces, let alone get out of its sandbox where it plays with drool all over its face. The marketing stunt seems to be: Portray it as potentially dangerous, as powerful things are inherently dangerous.
- IAmGraydon 2y agoIt’s good to see people are catching on to this. Hype = profit.
- freehorse 2y agoAnd the bing chat was trying to get a man to divorce his wife. Is this the level of "AI safety research" nowadays?
- emmanueloga_ 2y agoI hope people remember that in AI "oversight mechanisms" isn’t about preventing some "Skynet" scenario; it’s about ensuring these systems don’t malfunction. Current AI tech is far from being self-aware, it’s advanced math following patterns. It wouldn't be too crazy to see an LLM somehow generating the program: if (sensor.reading() > Math.random()) { launchMissiles();}, and then that code somehow becoming operational due to, well, oversight. But papers like these seems written to exploit the subject for hype and marketing.
- BiteCode_dev 2y agoWell, it's also trained on scifi corpus where the ai acts that way. So it'ls logical it's in the weights.