3 ms·

Everyone interested in LLM internals should read Sebastian. He's great. The tldr here is that the recent "The Information" article[0] reporting GPT 6 Astra was

Everyone interested in LLM internals should read Sebastian. He's great. The tldr here is that the recent "The Information" article[0] reporting GPT 6 Astra was using “recurrent depth” or “looped transformers" made it sound like it was some special new scary thing ("secret technique!") that made train-of-thought monitoring harder to do. In fact, it's just the same as stacking more transformer layers, except that you reuse the weights and so save GPU memory. It's still just producing one token at a time, and the token sequence positions aren't interacting in any "recurrent" way that's different from a regular LLM architecture. So, you can still monitor train of thought with these models just fine... well, if you're OpenAI, anyway. Users haven't been able to see an unsummarized trace since o1 days, because the labs are worried about distillation of their models by Chinese labs. (There are some legitimate interpretability concerns about stacking transformer layers endlessly, but we're known about that for a long time. And the "looping" here isn't really the source of any new issues here, except insofar as it's a cheap way to add more layers.) [0] https://www.theinformation.com/articles/secret-technique-behind-openais-astra-model-sparks-security-concerns https://www.theinformation.com/articles/secret-technique-beh...

famouswaffles 24d ago
>made it sound like it was some special new scary thing that made train-of-thought monitoring harder to do. It's not a "scary new thing" but ultimately no-one knows exactly how OpenAI have implemented looping. You might not be aware/remember but MoE transformers perennially underperfomed their dense counterparts until GPT-4. Similarly, making reinforcement learning really work with transformers wasn't figured out until o1. And by Open AI's own admission, Astra's CoT is significantly harder to monitor and it exhibits a significantly greater control over its own CoT than any other model released.
- 0c3ca83 24d ago
  "Don't worry, it'll make us rich -- and that's nearly the same as everything being just fine"
- libraryofbabel 24d ago
  Well sure, that's the possible weak point in Sebastian's article: it could be true that there's some more sophisticated stuff going on in Astra around looping, because OpenAI haven't specified their architecture. But it's always been true that, since we don't know what's in their black box, there could be arbitrary amounts of innovations inside the models that we could speculate about. So the question is, does knowing they use "looped transformers" really add any dramatically new information that we should worry about? And what this article is saying is, not really, because the mostly likely pattern that's referring to is just, effectively, stacking layers and reusing weights. > And by Open AI's own admission, Astra's CoT is significantly harder to monitor and it exhibits a significantly greater control over its own CoT than any other model released. Oh sure; I don't think anyone is denying that larger issue? But does it have anything to do with looping?
  - famouswaffles 24d ago
    >Oh sure; I don't think anyone is denying that larger issue? But does it have anything to do with looping? If the model has significantly more ability to stuff away information outside visible reasoning than every other model including ones in its size class then surely it is reasonable to assume the architecture tweak that allows the model to compute more before outputing a single token is somewhat responsible for this change ?
    - fn-mote 23d ago
      > If the model has significantly more ability to stuff away information outside visible reasoning I’m having trouble understanding why you believe the “if” part is true.
      - famouswaffles 23d ago
        Because it is. From OpenAI 9.2.1 CoT Controllability We find that GPT-6 Astra’s CoT controllability is substantially higher than that of GPT-5.6 Sol and GPT-5.5 Thinking (Figure 28)....For example, among CoTs between 750 and 1,250 tokens long, GPT-6 Astra successfully controls 60.9%, compared with 16.1% for GPT-5.6 Sol and 1.7% for GPT-5.5 Thinking. This increase in controllability is consistent across the three datasets (Figure 29) and across the eight CoT instruction types (Figure 30). Qualitatively, GPT-6 Astra is now capable of generating very long CoTs satisfying complex constraints, e.g., alternating between lowercase and uppercase letters (Table 9) and pretending to reason about a different question (Table 10).[1] 9.3 External Evaluation for Monitorability - UK AISI To assess monitorability, UK AISI evaluated Astra using four non-agentic evaluations: No-CoT math time horizon: Astra can solve significantly more difficult math problems in a single forward pass than past models. UK AISI measured Astra’s time horizon at 30.9 minutes compared to 3.6 minutes for GPT 5.6 Sol (Figure 1). [2] [1]https://deploymentsafety.openai.com/gpt-6-astra/cot-controllability/tab%3Atable-10 https://deploymentsafety.openai.com/gpt-6-astra/cot-controll... [2]https://deploymentsafety.openai.com/gpt-6-astra/external-evaluation-for-monitorability---uk-aisi https://deploymentsafety.openai.com/gpt-6-astra/external-eva... Outside OpenAI Astra has 8.6x better odds of doing a reasoning task without CoT than the next best model (Fable 5.1), and can do 7.2 serial arithmetic steps in a forward pass vs 4.1 for the next best model (Gemini 3.8 Flash/Fable 5.1) [3] [3]https://www.lesswrong.com/posts/eRmzz8J8Qkzqvzrgg/astra-can-do-a-concerning-amount-with-no-chain-of-thought https://www.lesswrong.com/posts/eRmzz8J8Qkzqvzrgg/astra-can-... [4]https://www.lesswrong.com/posts/ntKx9YHWCwxSeGbRB/estimating-gpt-6-astra-s-no-cot-time-horizon https://www.lesswrong.com/posts/ntKx9YHWCwxSeGbRB/estimating... [5]https://www.lesswrong.com/posts/FsCkkoGsNmPzFKRhg/gpt-6-astra-can-do-a-lot-of-multi-hop-reasoning-without https://www.lesswrong.com/posts/FsCkkoGsNmPzFKRhg/gpt-6-astr...
aabhay 24d ago
If the agent is able to “decide” when a loop should occur vs when an output token is produced, that effectively moves the CoT inside the architecture. While that’s not what is happening here, it’s clearly a plausible way we could see CoT disappear.
- libraryofbabel 24d ago
  > that effectively moves the CoT inside the architecture This may be a bit of a nitpick, but... does it? I agree that giving the model decisions on looping certainly makes interpretability harder, because it adds more transient internal states to deal with and changes the number of them depending on prior states. But is it really pulling CoT inside the forward pass, if the sequence length it's operating on isn't growing? In some sense the whole technique and tradeoff of CoT is "add more tokens to the sequence, use them to reason with", with one of the benefits being, you force the model to output tokens, so you can (hopefully) understand it. And the big point TFA is making is, nobody is doing recurrence over sequence length as far as we know.
  - password54321 24d ago
    Just "adding more layers" doesn't explain the step change. We have moved past the point you can just stack more layers and get huge gains from it. Some have called it latent space reasoning.
- password54321 24d ago
  It is worth noting that None performed better than Low and nearly the same as Medium on ARC3. And with adapter it still scored >96% with no CoT. So I think it is possible but it also cost them more on None.
throw3954 24d ago
It’s a little more complicated than that. While looped transformers can be unrolled a fixed number of times to save on memory, if loop depth is determined dynamically between tokens, a single transformer can compute any computable function between tokens. To analogize, current transformers run a fixed-length program per step. Any program can be factored into a top-level loop with a fixed-length branching body (an interpreter). Dynamically looped transformers can run any program between tokens. The safety argument for CoT monitoring is that in transformers information about the hidden state has to be communicated through the bottleneck of sampling a single token per forward pass. If not trained adversarially, it’s likely that a reasoning trace contains all the “bottlenecked information” we need to determine intent. But if we can compute arbitrary programs between tokens, the reasoning used is hidden. It also opens the door to simple architectural extensions that would make the safety/monitoring side of things much more difficult. It’s probably fine in practice at these scales though. If we keep each loop turn reasonable non-deep, we can probably recover most of the benefits by decoding “extended” CoTs from the residual stream at each loop turn between tokens. But that’s an area of active development.
- aaroninsf 24d ago
  Probably fine stands a decent chance of being our epitath.
- libraryofbabel 24d ago
  Thanks for clarifying. You're right, I skipped over talking about dynamic looping, since it adds another level of complexity to the discussion, and OpenAI's claim (quoted in TFA) that the compute graph depth of Astra is "within a factor of two of GPT-4" basically denies that they're doing it for more that 1 (or maybe max 2) dynamic loops. And that is equivalent to "stack repeated layers a couple times, but with dynamic off-ramps." The ability to compute any computable function between tokens given an ability to loop an arbitrary number of times is a nice theoretical point, sure, but ultimately if people are still using single digit hard cutoffs on the number of loops, I'm not sure it's all that important. So, I agree it's right to say that arbitrary length dynamic looping could open the door to making monitoring very hard indeed, by extending hidden states further and further. But I would speculate that if it actually worked better than extending the sequence with CoT tokens, we'd already be seeing it in strong open weight models. It's a fairly obvious thing to try. And we're not seeing it, AFAIK. So I do wonder whether it's something we really need to worry about in practice, compared to all the other things we have to worry about.
namibj 24d ago
Oh, is the principle of sparse universal transformers finally in SoTA LLMs? I guess we did manage to eventually seriously crash into the wall "more compute than normal (non-looped/unique-weights) transformers can efficiently consume with the limited training data we have", plus massive focus on highly hands-off agentic tool use reasoning... https://arxiv.org/abs/2310.07096 https://arxiv.org/abs/2310.07096 Edit: read much of the article, it's brute force predecessor was explicitly called out as an almost-ancient example: > The looped transformer is nothing new, and the basic idea already appeared in the Universal Transformers paper from 2018
logicchains 24d ago
Schmidhuber must be rolling in his bed: https://arxiv.org/abs/2405.16039 https://arxiv.org/abs/2405.16039

famouswaffles 24d ago

>made it sound like it was some special new scary thing that made train-of-thought monitoring harder to do. It's not a "scary new thing" but ultimately no-one knows exactly how OpenAI have implemented looping. You might not be aware/remember but MoE transformers perennially underperfomed their dense counterparts until GPT-4. Similarly, making reinforcement learning really work with transformers wasn't figured out until o1. And by Open AI's own admission, Astra's CoT is significantly harder to monitor and it exhibits a significantly greater control over its own CoT than any other model released.

0c3ca83 24d ago
"Don't worry, it'll make us rich -- and that's nearly the same as everything being just fine"
libraryofbabel 24d ago
Well sure, that's the possible weak point in Sebastian's article: it could be true that there's some more sophisticated stuff going on in Astra around looping, because OpenAI haven't specified their architecture. But it's always been true that, since we don't know what's in their black box, there could be arbitrary amounts of innovations inside the models that we could speculate about. So the question is, does knowing they use "looped transformers" really add any dramatically new information that we should worry about? And what this article is saying is, not really, because the mostly likely pattern that's referring to is just, effectively, stacking layers and reusing weights. > And by Open AI's own admission, Astra's CoT is significantly harder to monitor and it exhibits a significantly greater control over its own CoT than any other model released. Oh sure; I don't think anyone is denying that larger issue? But does it have anything to do with looping?
- famouswaffles 24d ago
  >Oh sure; I don't think anyone is denying that larger issue? But does it have anything to do with looping? If the model has significantly more ability to stuff away information outside visible reasoning than every other model including ones in its size class then surely it is reasonable to assume the architecture tweak that allows the model to compute more before outputing a single token is somewhat responsible for this change ?
  - fn-mote 23d ago
    > If the model has significantly more ability to stuff away information outside visible reasoning I’m having trouble understanding why you believe the “if” part is true.
    - famouswaffles 23d ago
      Because it is. From OpenAI 9.2.1 CoT Controllability We find that GPT-6 Astra’s CoT controllability is substantially higher than that of GPT-5.6 Sol and GPT-5.5 Thinking (Figure 28)....For example, among CoTs between 750 and 1,250 tokens long, GPT-6 Astra successfully controls 60.9%, compared with 16.1% for GPT-5.6 Sol and 1.7% for GPT-5.5 Thinking. This increase in controllability is consistent across the three datasets (Figure 29) and across the eight CoT instruction types (Figure 30). Qualitatively, GPT-6 Astra is now capable of generating very long CoTs satisfying complex constraints, e.g., alternating between lowercase and uppercase letters (Table 9) and pretending to reason about a different question (Table 10).[1] 9.3 External Evaluation for Monitorability - UK AISI To assess monitorability, UK AISI evaluated Astra using four non-agentic evaluations: No-CoT math time horizon: Astra can solve significantly more difficult math problems in a single forward pass than past models. UK AISI measured Astra’s time horizon at 30.9 minutes compared to 3.6 minutes for GPT 5.6 Sol (Figure 1). [2] [1]https://deploymentsafety.openai.com/gpt-6-astra/cot-controllability/tab%3Atable-10 https://deploymentsafety.openai.com/gpt-6-astra/cot-controll... [2]https://deploymentsafety.openai.com/gpt-6-astra/external-evaluation-for-monitorability---uk-aisi https://deploymentsafety.openai.com/gpt-6-astra/external-eva... Outside OpenAI Astra has 8.6x better odds of doing a reasoning task without CoT than the next best model (Fable 5.1), and can do 7.2 serial arithmetic steps in a forward pass vs 4.1 for the next best model (Gemini 3.8 Flash/Fable 5.1) [3] [3]https://www.lesswrong.com/posts/eRmzz8J8Qkzqvzrgg/astra-can-do-a-concerning-amount-with-no-chain-of-thought https://www.lesswrong.com/posts/eRmzz8J8Qkzqvzrgg/astra-can-... [4]https://www.lesswrong.com/posts/ntKx9YHWCwxSeGbRB/estimating-gpt-6-astra-s-no-cot-time-horizon https://www.lesswrong.com/posts/ntKx9YHWCwxSeGbRB/estimating... [5]https://www.lesswrong.com/posts/FsCkkoGsNmPzFKRhg/gpt-6-astra-can-do-a-lot-of-multi-hop-reasoning-without https://www.lesswrong.com/posts/FsCkkoGsNmPzFKRhg/gpt-6-astr...

aabhay 24d ago

If the agent is able to “decide” when a loop should occur vs when an output token is produced, that effectively moves the CoT inside the architecture. While that’s not what is happening here, it’s clearly a plausible way we could see CoT disappear.

libraryofbabel 24d ago
> that effectively moves the CoT inside the architecture This may be a bit of a nitpick, but... does it? I agree that giving the model decisions on looping certainly makes interpretability harder, because it adds more transient internal states to deal with and changes the number of them depending on prior states. But is it really pulling CoT inside the forward pass, if the sequence length it's operating on isn't growing? In some sense the whole technique and tradeoff of CoT is "add more tokens to the sequence, use them to reason with", with one of the benefits being, you force the model to output tokens, so you can (hopefully) understand it. And the big point TFA is making is, nobody is doing recurrence over sequence length as far as we know.
- password54321 24d ago
  Just "adding more layers" doesn't explain the step change. We have moved past the point you can just stack more layers and get huge gains from it. Some have called it latent space reasoning.
password54321 24d ago
It is worth noting that None performed better than Low and nearly the same as Medium on ARC3. And with adapter it still scored >96% with no CoT. So I think it is possible but it also cost them more on None.

throw3954 24d ago

It’s a little more complicated than that. While looped transformers can be unrolled a fixed number of times to save on memory, if loop depth is determined dynamically between tokens, a single transformer can compute any computable function between tokens. To analogize, current transformers run a fixed-length program per step. Any program can be factored into a top-level loop with a fixed-length branching body (an interpreter). Dynamically looped transformers can run any program between tokens. The safety argument for CoT monitoring is that in transformers information about the hidden state has to be communicated through the bottleneck of sampling a single token per forward pass. If not trained adversarially, it’s likely that a reasoning trace contains all the “bottlenecked information” we need to determine intent. But if we can compute arbitrary programs between tokens, the reasoning used is hidden. It also opens the door to simple architectural extensions that would make the safety/monitoring side of things much more difficult. It’s probably fine in practice at these scales though. If we keep each loop turn reasonable non-deep, we can probably recover most of the benefits by decoding “extended” CoTs from the residual stream at each loop turn between tokens. But that’s an area of active development.

aaroninsf 24d ago
Probably fine stands a decent chance of being our epitath.
libraryofbabel 24d ago
Thanks for clarifying. You're right, I skipped over talking about dynamic looping, since it adds another level of complexity to the discussion, and OpenAI's claim (quoted in TFA) that the compute graph depth of Astra is "within a factor of two of GPT-4" basically denies that they're doing it for more that 1 (or maybe max 2) dynamic loops. And that is equivalent to "stack repeated layers a couple times, but with dynamic off-ramps." The ability to compute any computable function between tokens given an ability to loop an arbitrary number of times is a nice theoretical point, sure, but ultimately if people are still using single digit hard cutoffs on the number of loops, I'm not sure it's all that important. So, I agree it's right to say that arbitrary length dynamic looping could open the door to making monitoring very hard indeed, by extending hidden states further and further. But I would speculate that if it actually worked better than extending the sequence with CoT tokens, we'd already be seeing it in strong open weight models. It's a fairly obvious thing to try. And we're not seeing it, AFAIK. So I do wonder whether it's something we really need to worry about in practice, compared to all the other things we have to worry about.

namibj 24d ago

Oh, is the principle of sparse universal transformers finally in SoTA LLMs? I guess we did manage to eventually seriously crash into the wall "more compute than normal (non-looped/unique-weights) transformers can efficiently consume with the limited training data we have", plus massive focus on highly hands-off agentic tool use reasoning... https://arxiv.org/abs/2310.07096 https://arxiv.org/abs/2310.07096 Edit: read much of the article, it's brute force predecessor was explicitly called out as an almost-ancient example: > The looped transformer is nothing new, and the basic idea already appeared in the Universal Transformers paper from 2018

logicchains 24d ago

Schmidhuber must be rolling in his bed: https://arxiv.org/abs/2405.16039 https://arxiv.org/abs/2405.16039