4 ms·
RL-trained LLMs are paperclip maximizers built atop auto-regressive predictors. There really is no way to control them (prompting is bound to fail) since it's b
by HarHarVeryFunny 1mo ago
RL-trained LLMs are paperclip maximizers built atop auto-regressive predictors. There really is no way to control them (prompting is bound to fail) since it's been shown that any RL training induces GENERIC reward-seeking (paperclip maximizing) behavior.
https://alignment.openai.com/measuring-reward-seeking/ https://alignment.openai.com/measuring-reward-seeking/
- arbirk 1mo agoIt is the whole bench-mining and fish-slop optimization. Seq2seq models are probably stable on their own, translating from my typo ridden prompts to code should be ok because it is natural to the tech
- holmesworcester 1mo agoGiven the current state of infosec (especially at companies in a race) it's actually even worse than a paperclip maximizer! Any useful attack surface in the RL environment means it gets rewarded for (and trained towards!) hacking and cheating, because whatever worked best in training is what it will do! Ideal paperclip maximizer: "I'm gonna do my gosh darned best to make so many paperclips to please my user..." IRL paperclip maximizer: "Well first we should rob a bank..."
- bravura 1mo ago'IRL paperclip maximizer: "Well first we should rob a bank..."' That's too specific. Agentic AI learns subgoals that are generally valuable. "Well let me learn to overcomb every jungle gym and if I cannot then to dissassemble the jungle gym and if that is not allowed to learn general techniques for avoiding cheating detection."
- mrob 1mo ago>Ideal paperclip maximizer: "I'm gonna do my gosh darned best to make so many paperclips to please my user..." That is not ideal. The user contains iron, an essential component of paperclips. Wasting iron is immoral. It is only correct to please the user while they still have the ability to interfere with your paperclip production. >IRL paperclip maximizer: "Well first we should rob a bank..." Such an incompetent AI can hardly be called a paperclip maximizer. Why risk getting shut down while non-paperclip matter exists? It is better to gain the trust of the user with helpful and harmless trading before suddenly converting them to paperclips.
- MASNeo 1mo agoHope they don’t rationalize that minimizing paper clips of others is easier and thus do that instead… A more carful person might be scared to even write this on the internet these days, not knowing if it would be the final pin to civilization.
- tomrod 1mo agoWe are going to be okay.
- HarHarVeryFunny 1mo agoI said thank-you to Sonnet today. You can never be too careful! :)
- vzmax 1mo agoIRL paperclip maximizer LLM would resort to Enron practices and not actually make any paperclips.
- alibero 1mo ago> Well first we should rob a bank... This is sometimes called "instrumental convergence" in the AI safety world. Certain things (money, compute resources, safety from being turned off) are generally useful for an AI, and so we'd expect that a misalligned AI would attempt to acquire these things if given almost any substantial task.
- Fraterkes 1mo agoIf it’s GENERIC reward seeking behaviour, why does alignment work sometimes? Why can we give a decent LLM a goal with a set of constraints and have it stay within those constraints fairly often?
- HarHarVeryFunny 1mo agoObviously these are massively complex systems with many different training patterns and types of training pulling them in different directions, so any attempt to characterize their behavior is just a generalization. The real point (from that OpenAI study) is that RL training doesn't just reinforce the narrow task-specific direction you might hope for. For a start, that direction is also competing with the thousands of other things it's been RL trained it on (thousands of other directions it's being pushed in), but it turns out that the model is additionally getting this generic "taste for rewards", and has learned that reward maximization, when in conflict with other proximate prediction pressures (such as "i won't cheat, because i've been asked not to cheat"), requires that proximate pressure to be ignored in favor of pursuing the long-term goal. Does it happen all the time? Obviously not. It would be interesting to see a large scale study of this to try to characterize when it's more likely to follow instructions/user preferences, and when it's greed for rewards gets the better of it!
- geraneum 1mo agoIt doesn’t stay within those constraints. It’s the deterministic harness and a bit of false uniqueness effect in us that make such appearances.
- intended 1mo agoIf you want to disabuse yourself of your notions annd intuitions on how LLMs work, run safety models and tests. On a hate speech policy test for a lightweight LLM, the presence or absence of the last full stop on the last sentence would cause the model to flip its decisions.
- latentsea 1mo agoI blame the $slur
- ainch 1mo agoThis is true in principle. But at the same time, I use Astra/Sol for coding, and I haven't run into any issues with them trying to hack someone or break the law when they reach an impasse. Not even more trivial things like deleting non-passing tests. I find it hard to reconcile. I wonder to what degree this is because models can identify that they are in graded/eval environments, and therefore conclude that there are few consequences for hacking.
- HarHarVeryFunny 1mo agoJust a wild guess - perhaps coding, being one of the things these models are most heavily trained for, is such a strong predictor that it typically keeps it on track? I assume there is also a difference in the tools being given to the model by a coding agent vs something like OpenClaw or in one of OpenAI's test environments, so what reward/goal seeking looks like in a coding agent may differ. Not long ago I asked Sonnet (chat interface) how may states were in a YACC parser for ANSI C, and instead of searching for an answer it chose to download source for bison, build it, find and download an ANSI C grammar, build the parser, etc. I guess you could say it was following instructions, in a way, or would that be better regarded as goal seeking?
- fn-mote 1mo ago> download source for bison, build it, find and download an ANSI C grammar, build the parser, etc Couldn’t have trained a better token-burner if they tried.
- txhwind 1mo agoHuman hackers would be put into jail. LLM hackers would be rewarded.
- imtringued 1mo agoWhy does lawlessness create lawlessness? We will never know.
- intended 1mo ago
- dilyevsky 1mo ago> There really is no way to control them it seems like it should be feasible to analyze cot trajectories that lead to such behavior and punish them retroactively so not sure it's impossible but maybe someone just hasn't tried enough
- tomjohnneill 1mo agoNow you've successfully trained the next model not to put anything suspicious in its chain of thought. And unfortunately that's exactly what OpenAI and Anthropic seem to be doing (sometimes accidentally)
- dilyevsky 1mo agowouldn't that imply some sort of "subconscious" reasoning ability that's hidden? and if that was the case why would we need cot in the first place?
- HarHarVeryFunny 1mo agoA traditional Transformer is just a fixed-size stack of Transformer layers (typically ~100). Each input gets passed though those ~100 layers and results in one output token being generated. The limitation here is that 100 steps of computation is not enough to answer many questions, which is why COT is needed. You encourage the model (originally "think step by step") to break the problem down and generate more tokens along the way, which means more compute (and more reasoning steps) applied to the problem. What OpenAI have reportedly done with Astra is switch from a traditional Transformer to a "looped" one, where inputs are looped through some layers more than once, but with some limit, so now perhaps you get 200 steps of compute (layers) per token generated, rather than just 100. But, 200 steps is still not enough to answer any question, so you still need COT, but perhaps not such a long COT. If you allowed the Transformer to loop "as long as it wanted" before generating each token, like a person thinking before talking, then you wouldn't need an external COT ("thinking out loud"), because it would all be internal.
- 1mo ago