4 ms·

I thought this was already widely known? From March last year: https://transformer-circuits.pub/2025/attribution-graphs/biology.html#dives-cot https://transfor

I thought this was already widely known? From March last year: https://transformer-circuits.pub/2025/attribution-graphs/biology.html#dives-cot https://transformer-circuits.pub/2025/attribution-graphs/bio... There's no reason to believe the model's self-reported "thinking" bears any relation to the mechanics by which it arrived at some output.

orbital-decay 1mo ago
It's... complicated. Yes, RL reward hacking makes it learn "bird language" and yes, reasoning traces can be misleading. However they also pretty clearly steer the final reply and not simply justify it, and can stay somewhat coherent and relevant with readability SFT and rewards. All these phenomenas coexist, they aren't mutually exclusive. Reasoning traces are still useful for debugging.
8note 1mo ago
that sounds testable - if you skip the reasoning tokens, do you get the same result? if not, then there's certainly some bearing, but not necessarily in how we read the tokens as text
- rcxdude 1mo ago
  It is and has been - skipping the tokens causes performance to drop. But replacing the tokens with filler causes the performance to drop, but by a lot less. You can also train models to emit broken or unrelated thinking tokens and their performance is also not much worse than the ones that are trained to output somewhat coherent thinking traces. This points to a hypothesis that the content of the tokens is only slightly related to the mechanism by which it improves performance, and that primarily the extra tokens allow the original prompt to be processed more deeply by the model, because earlier tokens will essentially pass through the model many more times than later ones.
  - neuroticnews25 1mo ago
    I'll never understand some downvoters here.
    - ForHackernews 1mo ago
      AI boosters denying that their machine god is burbling sweet nothings to itself?

orbital-decay 1mo ago

It's... complicated. Yes, RL reward hacking makes it learn "bird language" and yes, reasoning traces can be misleading. However they also pretty clearly steer the final reply and not simply justify it, and can stay somewhat coherent and relevant with readability SFT and rewards. All these phenomenas coexist, they aren't mutually exclusive. Reasoning traces are still useful for debugging.

8note 1mo ago

that sounds testable - if you skip the reasoning tokens, do you get the same result? if not, then there's certainly some bearing, but not necessarily in how we read the tokens as text

rcxdude 1mo ago
It is and has been - skipping the tokens causes performance to drop. But replacing the tokens with filler causes the performance to drop, but by a lot less. You can also train models to emit broken or unrelated thinking tokens and their performance is also not much worse than the ones that are trained to output somewhat coherent thinking traces. This points to a hypothesis that the content of the tokens is only slightly related to the mechanism by which it improves performance, and that primarily the extra tokens allow the original prompt to be processed more deeply by the model, because earlier tokens will essentially pass through the model many more times than later ones.
- neuroticnews25 1mo ago
  I'll never understand some downvoters here.
  - ForHackernews 1mo ago
    AI boosters denying that their machine god is burbling sweet nothings to itself?