3 ms·

It’s a little more complicated than that. While looped transformers can be unrolled a fixed number of times to save on memory, if loop depth is determined dynam

It’s a little more complicated than that. While looped transformers can be unrolled a fixed number of times to save on memory, if loop depth is determined dynamically between tokens, a single transformer can compute any computable function between tokens. To analogize, current transformers run a fixed-length program per step. Any program can be factored into a top-level loop with a fixed-length branching body (an interpreter). Dynamically looped transformers can run any program between tokens. The safety argument for CoT monitoring is that in transformers information about the hidden state has to be communicated through the bottleneck of sampling a single token per forward pass. If not trained adversarially, it’s likely that a reasoning trace contains all the “bottlenecked information” we need to determine intent. But if we can compute arbitrary programs between tokens, the reasoning used is hidden. It also opens the door to simple architectural extensions that would make the safety/monitoring side of things much more difficult. It’s probably fine in practice at these scales though. If we keep each loop turn reasonable non-deep, we can probably recover most of the benefits by decoding “extended” CoTs from the residual stream at each loop turn between tokens. But that’s an area of active development.

aaroninsf 23d ago
Probably fine stands a decent chance of being our epitath.
libraryofbabel 23d ago
Thanks for clarifying. You're right, I skipped over talking about dynamic looping, since it adds another level of complexity to the discussion, and OpenAI's claim (quoted in TFA) that the compute graph depth of Astra is "within a factor of two of GPT-4" basically denies that they're doing it for more that 1 (or maybe max 2) dynamic loops. And that is equivalent to "stack repeated layers a couple times, but with dynamic off-ramps." The ability to compute any computable function between tokens given an ability to loop an arbitrary number of times is a nice theoretical point, sure, but ultimately if people are still using single digit hard cutoffs on the number of loops, I'm not sure it's all that important. So, I agree it's right to say that arbitrary length dynamic looping could open the door to making monitoring very hard indeed, by extending hidden states further and further. But I would speculate that if it actually worked better than extending the sequence with CoT tokens, we'd already be seeing it in strong open weight models. It's a fairly obvious thing to try. And we're not seeing it, AFAIK. So I do wonder whether it's something we really need to worry about in practice, compared to all the other things we have to worry about.
WhitneyLand 23d ago
By that logic we should also consider the case of cutting the number of layers in half because that would also reduce hidden state between token generation. In your generalized example I think the concern is when the additional evaluation effectively becomes a replacement for CoT, where something like the coconut research could replace it completely. However, I don’t think we’re anywhere close to that with Astra.
sigbottle 23d ago
Nit: is it any computable function? I thought the requirements were unbounded (in principle) memory and time. (For all intents and purposes given how high dimensional you are and using the "vibes" of computability yes I agree w/ you)
program_whiz 23d ago
Actually, removing CoT might make models safer, because we can analyze the entire landscape of their potential outputs, rather than a point-sample (we'll never know how close we were to "kill all humans"). By inspecting intermediate vector spaces, we can actually get certainty bounds on how safely the model is behaving (or even trending). Wrote about it here: https://substack.com/home/post/p-214402969 https://substack.com/home/post/p-214402969
- fc417fc802 23d ago
  I don't see why you have to remove CoT to do that?
  - program_whiz 23d ago
    Good point, you don't have to -- but my argument is just that removing CoT doesn't make things less safe. Anything CoT can tell you is just a point sample of a probability surface. Having the whole probability surface can already answer any question the point sample can answer (for example, how likely is the model to produce a problematic phrase). While its more computationally expensive, you could always just draw point samples like the model does and evaluate those (or use temperature zero to just sample the most likely output tokens).
    - AS04 22d ago
      No, this argument doesn't make any sense. With CoT, the model must compress hidden state to actual text and use its scratchpad as a bottlenecked representation of its past thinking. Thus, it is observable and we can tell by the pattern of a CoT what it was thinking to some extent if we do proper interpretability. Change the CoT text, and the model has literally changed the way it was thinking for the next tokens. How would you do the same if all reasoning is happening in looped transformers? You would have to develop very sophisticated interventions that construct hidden states which you inject into the model instead while it is thinking. Much harder, and much easier for the model to use weird correlations across the hidden state to hide misaligned thought patterns.
imtringued 23d ago
>It’s a little more complicated than that. While looped transformers can be unrolled a fixed number of times to save on memory, if loop depth is determined dynamically between tokens, a single transformer can compute any computable function between tokens. This is worded so confusingly it might as well tell us nothing, because it is technically true even without looping due to the fact that you still have infinitely growing context and can simulate a standard turing machine using it. If you loop, you have a fixed capacity memory that you can rewrite but not carry over to the next token, this is different from a non looped transformer where the transformer can only append a new token. Meanwhile if you have a DEQ with growing context, it is bona-fide turing complete in the most literal sense.

aaroninsf 23d ago

Probably fine stands a decent chance of being our epitath.

libraryofbabel 23d ago

Thanks for clarifying. You're right, I skipped over talking about dynamic looping, since it adds another level of complexity to the discussion, and OpenAI's claim (quoted in TFA) that the compute graph depth of Astra is "within a factor of two of GPT-4" basically denies that they're doing it for more that 1 (or maybe max 2) dynamic loops. And that is equivalent to "stack repeated layers a couple times, but with dynamic off-ramps." The ability to compute any computable function between tokens given an ability to loop an arbitrary number of times is a nice theoretical point, sure, but ultimately if people are still using single digit hard cutoffs on the number of loops, I'm not sure it's all that important. So, I agree it's right to say that arbitrary length dynamic looping could open the door to making monitoring very hard indeed, by extending hidden states further and further. But I would speculate that if it actually worked better than extending the sequence with CoT tokens, we'd already be seeing it in strong open weight models. It's a fairly obvious thing to try. And we're not seeing it, AFAIK. So I do wonder whether it's something we really need to worry about in practice, compared to all the other things we have to worry about.

WhitneyLand 23d ago

By that logic we should also consider the case of cutting the number of layers in half because that would also reduce hidden state between token generation. In your generalized example I think the concern is when the additional evaluation effectively becomes a replacement for CoT, where something like the coconut research could replace it completely. However, I don’t think we’re anywhere close to that with Astra.

sigbottle 23d ago

Nit: is it any computable function? I thought the requirements were unbounded (in principle) memory and time. (For all intents and purposes given how high dimensional you are and using the "vibes" of computability yes I agree w/ you)

program_whiz 23d ago

Actually, removing CoT might make models safer, because we can analyze the entire landscape of their potential outputs, rather than a point-sample (we'll never know how close we were to "kill all humans"). By inspecting intermediate vector spaces, we can actually get certainty bounds on how safely the model is behaving (or even trending). Wrote about it here: https://substack.com/home/post/p-214402969 https://substack.com/home/post/p-214402969

fc417fc802 23d ago
I don't see why you have to remove CoT to do that?
- program_whiz 23d ago
  Good point, you don't have to -- but my argument is just that removing CoT doesn't make things less safe. Anything CoT can tell you is just a point sample of a probability surface. Having the whole probability surface can already answer any question the point sample can answer (for example, how likely is the model to produce a problematic phrase). While its more computationally expensive, you could always just draw point samples like the model does and evaluate those (or use temperature zero to just sample the most likely output tokens).
  - AS04 22d ago
    No, this argument doesn't make any sense. With CoT, the model must compress hidden state to actual text and use its scratchpad as a bottlenecked representation of its past thinking. Thus, it is observable and we can tell by the pattern of a CoT what it was thinking to some extent if we do proper interpretability. Change the CoT text, and the model has literally changed the way it was thinking for the next tokens. How would you do the same if all reasoning is happening in looped transformers? You would have to develop very sophisticated interventions that construct hidden states which you inject into the model instead while it is thinking. Much harder, and much easier for the model to use weird correlations across the hidden state to hide misaligned thought patterns.

imtringued 23d ago

>It’s a little more complicated than that. While looped transformers can be unrolled a fixed number of times to save on memory, if loop depth is determined dynamically between tokens, a single transformer can compute any computable function between tokens. This is worded so confusingly it might as well tell us nothing, because it is technically true even without looping due to the fact that you still have infinitely growing context and can simulate a standard turing machine using it. If you loop, you have a fixed capacity memory that you can rewrite but not carry over to the next token, this is different from a non looped transformer where the transformer can only append a new token. Meanwhile if you have a DEQ with growing context, it is bona-fide turing complete in the most literal sense.