4 ms·

I always wondered what the model meant when it writes "I'm now considering the architecture of the service" but outputs nothing of the sorts in its CoT. Is the

I always wondered what the model meant when it writes "I'm now considering the architecture of the service" but outputs nothing of the sorts in its CoT. Is the model really "thinking" about that stuff or is just mimicking human "manners"? And if so, where the thinking is happening if it is not in the literal chain of *thought*? I'm not sure J-Space is the answer to that question, but very interesting nevertheless.

wongarsu 3mo ago
Almost none of the hosted models give you their unredacted CoT. Claude certainly doesn't, what you get are fragments and summaries from it. There are various justifications on this, but it's mostly to make distillation and fine tuning off their model outputs a bit harder for their competitors
- aabhay 3mo ago
  Its also because the CoT is probably unintelligible
  - optimalsolver 3mo ago
    More legible than seems at first glance: https://www.lesswrong.com/posts/wCSEpT3dTGz4N86Wi/even-illegible-mythos-reasoning-traces-seem-pretty-legible https://www.lesswrong.com/posts/wCSEpT3dTGz4N86Wi/even-illeg...
    - orbital-decay 3mo ago
      The text can be legible, but the meaning the model assigns to these words can be subtly different, and you have no way to tell. This is evident when trying to make most modern reasoning models follow a fixed CoT plan by filling a form with placeholders, they're extremely stubborn because they simply don't understand your words in it! They learned their own language in their CoT. Sometimes they're controlled for readability during training but even then they find a way to circumvent this, for example Gemini 3.0 had perfectly readable raw CoT but was barely able to follow such a plan (3.5 Flash is way better at following, they clearly improved it).
baq 3mo ago
> I'm now considering the architecture of the service What you see here is a summary of thinking tokens written by some other smaller model (e.g. old sonnet). The actual thinking sometimes (rarely) leaks and is not easy to parse.
andrewlin247 3mo ago
> Is the model really "thinking" about that stuff or is just mimicking human "manners"? Well, what's the difference? If it's pretending to think and its thoughts correlate to its final output, then I'd say that really is thinking.
ACCount37 3mo ago
The answer is, as is often the case, "yes". In some cases, an LLM may truly "consider the architecture" internally, within its latent representations, and in others, it can output a similar phrase simply because it's "expected" of it. "Where" is pretty clear. There aren't that many places within an LLM, and hidden state is the main culprit. How to read that space is another matter entirely.

wongarsu 3mo ago

Almost none of the hosted models give you their unredacted CoT. Claude certainly doesn't, what you get are fragments and summaries from it. There are various justifications on this, but it's mostly to make distillation and fine tuning off their model outputs a bit harder for their competitors

aabhay 3mo ago
Its also because the CoT is probably unintelligible
- optimalsolver 3mo ago
  More legible than seems at first glance: https://www.lesswrong.com/posts/wCSEpT3dTGz4N86Wi/even-illegible-mythos-reasoning-traces-seem-pretty-legible https://www.lesswrong.com/posts/wCSEpT3dTGz4N86Wi/even-illeg...
  - orbital-decay 3mo ago
    The text can be legible, but the meaning the model assigns to these words can be subtly different, and you have no way to tell. This is evident when trying to make most modern reasoning models follow a fixed CoT plan by filling a form with placeholders, they're extremely stubborn because they simply don't understand your words in it! They learned their own language in their CoT. Sometimes they're controlled for readability during training but even then they find a way to circumvent this, for example Gemini 3.0 had perfectly readable raw CoT but was barely able to follow such a plan (3.5 Flash is way better at following, they clearly improved it).

baq 3mo ago

> I'm now considering the architecture of the service What you see here is a summary of thinking tokens written by some other smaller model (e.g. old sonnet). The actual thinking sometimes (rarely) leaks and is not easy to parse.

andrewlin247 3mo ago

> Is the model really "thinking" about that stuff or is just mimicking human "manners"? Well, what's the difference? If it's pretending to think and its thoughts correlate to its final output, then I'd say that really is thinking.

ACCount37 3mo ago

The answer is, as is often the case, "yes". In some cases, an LLM may truly "consider the architecture" internally, within its latent representations, and in others, it can output a similar phrase simply because it's "expected" of it. "Where" is pretty clear. There aren't that many places within an LLM, and hidden state is the main culprit. How to read that space is another matter entirely.