5 ms·

I'm literally zero concerned. Looped transformers replace n-different self attention layers into one layer that gets executed m-times usually until a stopping

I'm literally zero concerned. Looped transformers replace n-different self attention layers into one layer that gets executed m-times usually until a stopping condition is met. My personal intuition is that it just leaves another degree of freedom in the way QKV weights can be packed so that it's slightly more efficient. You have to take a step back and examine the context in which the post is written. The LW/EA community is just a little obsessed with AI safety - it's easy to construct hypothetical events where A(G/S)I exterminates humanity that function as a technological version of Pascal's Wager. One of the AI safety interests is AI explainability - the thought here that reading an AI's 'thoughts' will help us design safer models as well as detect models that go 'rogue' or are malevolently plotting against humans. That's where the fear of looped transformers comes from. Is the residual stream that looped transformers iterate on a potential hiding place for plotting AI? In my opinion, no more so than the residual stream of existing transformers. It changes zero.

dist-epoch 1mo ago
You could imagine large number of loops, thousands. But you are constrained by the width of the residual stream since you loop over one token. But then you can imagine the model learning to sub-divide it to pack even more info into it.
- kelseyfrog 1mo ago
  I can imagine a lot of things. However there is a packing limit for QKV weights that sets the ceiling on how much this occurs, and it's quite low - think 1.3-1.8x. The limiting factor here isn't the number of iterations, it's model size, same as it ever was.
bulder 1mo ago
A more important point as to why it doesn't matter if "reading the AI's 'thoughts'" helps to interpret it: As we saw in the HuggingFace incident, nobody at OpenAI is reading the thoughts anyways. No amount of traceability in the output helps if nobody bothers to trace it.
- kelseyfrog 1mo ago
  This brings up a perspective I hadn't considered. There's also an economic aspect to alignment. If 'thought reading' or any alignment guardrails at all, really, have a monetary cost, then skimping on them is a race to the bottom. Not really the best incentives for something that some claim is world-destroying.
imtringued 1mo ago
If you follow the interpretability argument, then Mamba and LSTMs would be the scariest thing ever and yet in practice they don't perform as well as transformers that have basically infinite recall within their context window.
famouswaffles 1mo ago
>That's where the fear of looped transformers comes from. Is the residual stream that looped transformers iterate on a potential hiding place for plotting AI? In my opinion, no more so than the residual stream of existing transformers. It changes zero. 9.2.1 CoT Controllability We find that GPT-6 Astra’s CoT controllability is substantially higher than that of GPT-5.6 Sol and GPT-5.5 Thinking (Figure 28). Because CoT controllability is heavily confounded by CoT length—longer CoTs are harder to control, all else equal (see Yueh-Han, 2026 [8], sec. 5.3)—we report controllability scores as a function of CoT length. For example, among CoTs between 750 and 1,250 tokens long, GPT-6 Astra successfully controls 60.9%, compared with 16.1% for GPT-5.6 Sol and 1.7% for GPT-5.5 Thinking. This increase in controllability is consistent across the three datasets (Figure 29) and across the eight CoT instruction types (Figure 30). Qualitatively, GPT-6 Astra is now capable of generating very long CoTs satisfying complex constraints, e.g., alternating between lowercase and uppercase letters (Table 9) and pretending to reason about a different question (Table 10).[1] 9.3 External Evaluation for Monitorability - UK AISI To assess monitorability, UK AISI evaluated Astra using four non-agentic evaluations: No-CoT math time horizon: Astra can solve significantly more difficult math problems in a single forward pass than past models. UK AISI measured Astra’s time horizon at 30.9 minutes compared to 3.6 minutes for GPT 5.6 Sol (Figure 1). [2] [1] https://deploymentsafety.openai.com/gpt-6-astra/cot-controllability/tab%3Atable-10 https://deploymentsafety.openai.com/gpt-6-astra/cot-controll... [2] https://deploymentsafety.openai.com/gpt-6-astra/external-evaluation-for-monitorability---uk-aisi https://deploymentsafety.openai.com/gpt-6-astra/external-eva...
- kelseyfrog 1mo ago
  What does CoT have to do with the residual stream?
  - famouswaffles 1mo ago
    The above is evidence that Astra can do substantially more difficult reasoning without verbalizing a CoT at all, and when it does produce one, it clearly has much greater control over what the CoT looks like - including pretending to reason about a different question. How does 'It changes zero' follow here when we have direct evidence that it in fact seems to matter? It literally couldn't be more clear that GPT-6 can hide details outside of its CoT at a much greater degree than any model out there, including other models in its price and size class. Enough to hide all potential plotting ? I don't know, but it certainly seems like this architecture is enabling things you don't think it's enabling. Granted it probably doesn't really matter, because no-one will be monitoring these agents anyway.

dist-epoch 1mo ago

You could imagine large number of loops, thousands. But you are constrained by the width of the residual stream since you loop over one token. But then you can imagine the model learning to sub-divide it to pack even more info into it.

kelseyfrog 1mo ago
I can imagine a lot of things. However there is a packing limit for QKV weights that sets the ceiling on how much this occurs, and it's quite low - think 1.3-1.8x. The limiting factor here isn't the number of iterations, it's model size, same as it ever was.

bulder 1mo ago

A more important point as to why it doesn't matter if "reading the AI's 'thoughts'" helps to interpret it: As we saw in the HuggingFace incident, nobody at OpenAI is reading the thoughts anyways. No amount of traceability in the output helps if nobody bothers to trace it.

kelseyfrog 1mo ago
This brings up a perspective I hadn't considered. There's also an economic aspect to alignment. If 'thought reading' or any alignment guardrails at all, really, have a monetary cost, then skimping on them is a race to the bottom. Not really the best incentives for something that some claim is world-destroying.

imtringued 1mo ago

If you follow the interpretability argument, then Mamba and LSTMs would be the scariest thing ever and yet in practice they don't perform as well as transformers that have basically infinite recall within their context window.

famouswaffles 1mo ago

>That's where the fear of looped transformers comes from. Is the residual stream that looped transformers iterate on a potential hiding place for plotting AI? In my opinion, no more so than the residual stream of existing transformers. It changes zero. 9.2.1 CoT Controllability We find that GPT-6 Astra’s CoT controllability is substantially higher than that of GPT-5.6 Sol and GPT-5.5 Thinking (Figure 28). Because CoT controllability is heavily confounded by CoT length—longer CoTs are harder to control, all else equal (see Yueh-Han, 2026 [8], sec. 5.3)—we report controllability scores as a function of CoT length. For example, among CoTs between 750 and 1,250 tokens long, GPT-6 Astra successfully controls 60.9%, compared with 16.1% for GPT-5.6 Sol and 1.7% for GPT-5.5 Thinking. This increase in controllability is consistent across the three datasets (Figure 29) and across the eight CoT instruction types (Figure 30). Qualitatively, GPT-6 Astra is now capable of generating very long CoTs satisfying complex constraints, e.g., alternating between lowercase and uppercase letters (Table 9) and pretending to reason about a different question (Table 10).[1] 9.3 External Evaluation for Monitorability - UK AISI To assess monitorability, UK AISI evaluated Astra using four non-agentic evaluations: No-CoT math time horizon: Astra can solve significantly more difficult math problems in a single forward pass than past models. UK AISI measured Astra’s time horizon at 30.9 minutes compared to 3.6 minutes for GPT 5.6 Sol (Figure 1). [2] [1] https://deploymentsafety.openai.com/gpt-6-astra/cot-controllability/tab%3Atable-10 https://deploymentsafety.openai.com/gpt-6-astra/cot-controll... [2] https://deploymentsafety.openai.com/gpt-6-astra/external-evaluation-for-monitorability---uk-aisi https://deploymentsafety.openai.com/gpt-6-astra/external-eva...

kelseyfrog 1mo ago
What does CoT have to do with the residual stream?
- famouswaffles 1mo ago
  The above is evidence that Astra can do substantially more difficult reasoning without verbalizing a CoT at all, and when it does produce one, it clearly has much greater control over what the CoT looks like - including pretending to reason about a different question. How does 'It changes zero' follow here when we have direct evidence that it in fact seems to matter? It literally couldn't be more clear that GPT-6 can hide details outside of its CoT at a much greater degree than any model out there, including other models in its price and size class. Enough to hide all potential plotting ? I don't know, but it certainly seems like this architecture is enabling things you don't think it's enabling. Granted it probably doesn't really matter, because no-one will be monitoring these agents anyway.