3 ms·

The paper isn't proposing "monitor chain of thought for safety" - that's already standard practice. The core finding is that training on CoT monitors destroys t

by ctoth 1y ago

The paper isn't proposing "monitor chain of thought for safety" - that's already standard practice. The core finding is that training on CoT monitors destroys them. They provide empirical evidence that doing the obvious thing - training models not to have "bad thoughts" - causes them to learn obfuscated reward hacking instead. The paper demonstrates that current models are already performing complex reward hacks in production environments, and that attempts to fix this via CoT training make the problem worse, not better. As for your "40 authors" snark - this is a position paper where researchers from competing labs (OpenAI, Anthropic, DeepMind, government safety institutes) are jointly committing to NOT do something that's locally tempting but globally catastrophic. Getting industry consensus on "don't train away bad thoughts even though it would make your models look safer" is the opposite of trivial. This reads like someone who saw a medical consensus statement saying "this common treatment kills patients" and responded with "did one person discover medicine exists and everyone else just agreed?"

latentsea 1y ago
> They provide empirical evidence that doing the obvious thing - training models not to have "bad thoughts" - causes them to learn obfuscated reward hacking instead Just like real humans.
the8472 1y ago
The core finding was already predicted[0], there have been previous papers[1], and kind of obvious considering RL systems have been specification-gaming for decades[2]. But yes, it's good to see broad agreement on it. [0] https://thezvi.substack.com/p/ai-68-remarkably-reasonable-reactions?open=false#%C2%A7the-art-of-the-jailbreak https://thezvi.substack.com/p/ai-68-remarkably-reasonable-re... [1] https://arxiv.org/abs/2503.11926 https://arxiv.org/abs/2503.11926 [2] https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOa...
- Turn_Trout 1y ago
  As someone who did their PhD in RL and alignment, it was not obvious to me a priori if, or when, or how badly obfuscation would be a problem. Yes, it's been predicted (and was predicted significantly before that Zvi post). But many other alignment fears have been _predicted_, and those didn't actually happen. I don't think the existence of specification gaming in unrelated settings was strong evidence that obfuscation would occur in modern CoT supervision. Speculatively, I think CoT obfuscation happens due to the internal structure of LLMs and it being inductively "easier" to reweight model circuits to not admit wrongthink, rather than to rewire circuits to solve problems in entirely different ways.
jerf 1y ago
I couldn't pull the link up easily, since the search terms are pretty jammed, but HN had a link to paper a couple of months back about how someone had the LLM do some basic arithmetic and report with a chain-of-thought as to how it was doing it. They then went directly into the neural net and decoded what it was actually doing in order to do the math. The two things were not the same. The chain-of-thought gave a reasonably "elementary school" explanation of how to do the addition, but what the model basically did in English was just intuit the answer in much the same way that we humans do not go through a process to figure out what "4 + 7" is... we pretty much just have neurons that just "know" the answer to that. (That's not exactly what happened but it's close enough for this post.) If CoT improves performance, then CoT improves performance, however the naively obvious read of "it improves performance because it is 'thinking' the 'thoughts' it tell us it is thinking, for the reasons it gives" is not completely accurate. It may not be completely wrong, either, but it's definitely not completely accurate. Given that I see no reason to believe it would be hard in the slightest to train models that have even more divergence between their "actual" thought processes and what they claim they are.
- antonvs 1y ago
  > If CoT improves performance, then CoT improves performance, however the naively obvious read of "it improves performance because it is 'thinking' the 'thoughts' it tell us it is thinking, for the reasons it gives" is not completely accurate. I can't imagine why anyone who knows even a little about how these models work would believe otherwise. The "chain of thought" is text generated by the model in response to a prompt, just like any other text it generates. It then consumes that as part of a new prompt, and generates more text. Those "thoughts" are obviously going to have an effect on the generated output, simply by virtue of being present in the prompt. And the evidence shows that it can help improve the quality of output. But there's no reason to expect that the generated "thoughts" would correlate directly or precisely with what's going on inside the model when it's producing text.
- nyrikki 1y ago
  Subbarao Kambhampati and his team has several papers strongly suggesting that the intermediate tokens are of no symantic import.[0] My hunch is that training explicitly for human acceptance of the intermediate/scratch tokens will end up being expensive. What you need is to converge to the correct answer, and scratch space dramatically improves expressivity[1], but the relationship to learning and retrieval is still open unless I missed something. Remember even access to a pure random Oracle dramatically improves compute time on some problems. [0] https://arxiv.org/abs/2505.13775 https://arxiv.org/abs/2505.13775 [1] https://arxiv.org/abs/2502.02393 https://arxiv.org/abs/2502.02393

by ctoth 1y ago

latentsea 1y ago

> They provide empirical evidence that doing the obvious thing - training models not to have "bad thoughts" - causes them to learn obfuscated reward hacking instead Just like real humans.

the8472 1y ago

The core finding was already predicted[0], there have been previous papers[1], and kind of obvious considering RL systems have been specification-gaming for decades[2]. But yes, it's good to see broad agreement on it. [0] https://thezvi.substack.com/p/ai-68-remarkably-reasonable-reactions?open=false#%C2%A7the-art-of-the-jailbreak https://thezvi.substack.com/p/ai-68-remarkably-reasonable-re... [1] https://arxiv.org/abs/2503.11926 https://arxiv.org/abs/2503.11926 [2] https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOa...

Turn_Trout 1y ago
As someone who did their PhD in RL and alignment, it was not obvious to me a priori if, or when, or how badly obfuscation would be a problem. Yes, it's been predicted (and was predicted significantly before that Zvi post). But many other alignment fears have been _predicted_, and those didn't actually happen. I don't think the existence of specification gaming in unrelated settings was strong evidence that obfuscation would occur in modern CoT supervision. Speculatively, I think CoT obfuscation happens due to the internal structure of LLMs and it being inductively "easier" to reweight model circuits to not admit wrongthink, rather than to rewire circuits to solve problems in entirely different ways.

jerf 1y ago

I couldn't pull the link up easily, since the search terms are pretty jammed, but HN had a link to paper a couple of months back about how someone had the LLM do some basic arithmetic and report with a chain-of-thought as to how it was doing it. They then went directly into the neural net and decoded what it was actually doing in order to do the math. The two things were not the same. The chain-of-thought gave a reasonably "elementary school" explanation of how to do the addition, but what the model basically did in English was just intuit the answer in much the same way that we humans do not go through a process to figure out what "4 + 7" is... we pretty much just have neurons that just "know" the answer to that. (That's not exactly what happened but it's close enough for this post.) If CoT improves performance, then CoT improves performance, however the naively obvious read of "it improves performance because it is 'thinking' the 'thoughts' it tell us it is thinking, for the reasons it gives" is not completely accurate. It may not be completely wrong, either, but it's definitely not completely accurate. Given that I see no reason to believe it would be hard in the slightest to train models that have even more divergence between their "actual" thought processes and what they claim they are.

antonvs 1y ago
> If CoT improves performance, then CoT improves performance, however the naively obvious read of "it improves performance because it is 'thinking' the 'thoughts' it tell us it is thinking, for the reasons it gives" is not completely accurate. I can't imagine why anyone who knows even a little about how these models work would believe otherwise. The "chain of thought" is text generated by the model in response to a prompt, just like any other text it generates. It then consumes that as part of a new prompt, and generates more text. Those "thoughts" are obviously going to have an effect on the generated output, simply by virtue of being present in the prompt. And the evidence shows that it can help improve the quality of output. But there's no reason to expect that the generated "thoughts" would correlate directly or precisely with what's going on inside the model when it's producing text.
nyrikki 1y ago
Subbarao Kambhampati and his team has several papers strongly suggesting that the intermediate tokens are of no symantic import.[0] My hunch is that training explicitly for human acceptance of the intermediate/scratch tokens will end up being expensive. What you need is to converge to the correct answer, and scratch space dramatically improves expressivity[1], but the relationship to learning and retrieval is still open unless I missed something. Remember even access to a pure random Oracle dramatically improves compute time on some problems. [0] https://arxiv.org/abs/2505.13775 https://arxiv.org/abs/2505.13775 [1] https://arxiv.org/abs/2502.02393 https://arxiv.org/abs/2502.02393