6 ms·
How concerned should we be about Astra's recurrent architecture?
- d_silin 1mo agoA debate between grossly incompetent against grossly immoral, honestly. You should ignore anything LessWrong or OpenAI says and do your own research.
- d_silin 1mo agoTo add a bit more constructive feedback, think of the "AI cornucopia" and "Superintelligence destroys humanity" as opposite ends of all possible outcomes distribution (low probability event). The most likely one is the "business as usual, but with AI" - some things will get better, some things will get worse, but overall state of affairs will remain mostly the same.
- elteto 1mo agoBut will “business as usual, but with AI” justify the current capital expenditures? I think the market is pricing things as being closer to “AI cornucopia”. What happens if/when we don’t deliver?
- d_silin 1mo agoMarket bubble will pop, not the first time and not the last, most likely after OpenAI and Anthropic IPOs.
- dgellow 1mo agoMy guess is after Anthropic and before OpenAI
- dgellow 1mo ago> What happens if/when we don’t deliver? I wish we had real journalism, the AI labs CEO should be asked that question in every single interview
- holmesworcester 1mo agoThe people who've thought the most about this put it differently: Think of a new, superintelligent model as if it was a new v1 Starship launching for the first time, with a full fuel tank. On the one hand, rockets have existed for some time, and some have gone to space successfully, including by this company. On the other hand, this is a tube of metal full of highly explosive liquid going faster than most human objects ever go, for the first time ever in this novel and state of the art configuration. If someone said, "really, the first Starship exploding is just at one end of the probability distribution, where the other is that everything goes fine and all its passengers have a nice trip in space," would you get on that rocket? Or, more aptly, if you and every other living human was already on that rocket, would you push the launch button? The analogy works because superintelligence is, like rocket fuel, an extremely powerful force that has a default tendency to break containment and go boom (consume lots of energy and heat and matter in a chain reaction, to pursue more intelligence to pursue whatever goal it is pursuing.)
- d_silin 1mo agoCurrent AIs are closer to bottle rockets than to the Starship on the intelligence scale. Some property damage already happened, but you can't master the art of rocketry without trial and error.
- stillpointlab 1mo ago> superintelligence is, like rocket fuel, an extremely powerful force that has a default tendency to break containment and go boom what evidence do we have this is the case?
- ForHackernews 1mo agoWe can learn from history: Albert Einstein famously tricked humanity into building nuclear weapons for him and was only prevented from wiping out all sentient life by the Princeton IAS Board of Alignment who published a very compelling blog post about realigning the A-bomb contra paperclips.
- 1mo ago
- samrus 1mo agoI like the idea of more reccurance in the transformer level. Chain of thought always seemed so clunky. Its just not the way the human brain processes information. Its an extrmeely crude approximation at best
- sznio 1mo agoit is what I do to solve hard problems through. easy stuff happens by itself, but with a system large enough you need a scratchpad and a rubber duck.
- dgellow 1mo agoOne thing about the reasoning is that models are trained to generate a chain of thoughts, but it doesn’t have to be correct, accurate, or reflect the underlying logic of the LLM. It’s the same problem we have with the output, it is something plausible, but not that reliable
- anon291 1mo agoI do the same thing in my head. There is no underlying logic to an llm. Logic is an external construct alien to human like forms of reasoning.
- trhway 1mo ago>but it doesn’t have to be correct, accurate, or ... why we don't do GAN here, ie. second model verifying correctness/accuracy/etc. ?
- naasking 1mo agoYes, both the output should be "milestones" of sorts, like lemmas and theorems in math. Important plateaus that serve as a launching pad to the next phase. Regurgitating every thought potentially degrades signal:noise ratio.
- anon291 1mo agoThe hidden states of the tokens likely contain more semantic information than can be extracted by the final projection into token space.
- dang 1mo agoRelated ongoing thread: OpenAI's new reasoning technique alarms AI safety experts - https://news.ycombinator.com/item?id=49552395 https://news.ycombinator.com/item?id=49552395
- smcg 1mo ago[flagged]
- Legend2440 1mo ago>This suggests that deeper isn't always better for looped transformers, which leaves me less worried about a race to the bottom toward looped transformers with hundreds of recurrent loops. I disagree with this. Deeper will always be at least as good because the extra loops can exit early or just no-op. Any performance degradation they're seeing at higher loop counts today is merely training stability issues, which can be overcome. Deeper almost certainly is better, and we will probably see not just hundreds but millions of recurrent loops in the future.
- nomel 1mo agoDeeper independent, sure. Deeper shared though? Information and signal theory still apply here. At infinite cycles, without new input, you'll end up with a locked state or oscillations. Some point before that, any "attractors" in the latent space, with slightly higher statistics, will pull things towards a space that might eventually be only loosely related to the goal, because each loop would be lossy, right?
- Legend2440 1mo ago>At infinite cycles, without new input, you'll end up with a locked state or oscillations. I don't think that's true; there are computations that take infinite steps but never converge or repeat, like the mandelbrot set. Looping for millions or billions of steps is absolutely normal in traditional algorithms. We know from complexity theory that some computations require a minimum number of steps. More depth is just more room for computation.
- pennomi 1mo agoOr even more simply, calculating the digits of pi is an infinite number of steps.
- nomel 1mo agoSure, but I don't think that's related to what you're implying, which is that, correct me if I'm wrong, the same number of weight should be able to hold many orders of magnitude more information by being reused. These are not deterministic functions or systems that have infinite precision. See the "should I drive or walk my car to the car wash", or any of the other logic riddle problems, for examples of a statistical attractors.
- kjshsh123 1mo ago>In contrast to a classic RNN, there's no unbounded hidden state accumulating across an entire trajectory I don' understand this line. In a classic RNN hidden state is bounded dimension. In fact it's transformers that technically have unbounded hidden state. You can't parallelize classic nonlinear RNNs for various reasons but in training both RNN and Transformer depend on the entire sequence history in a way that is unbounded. Of course in practice you just train on a max sequence length. RNN xhat[t+1]=f(x[t],h[t]) Transformer/self-attention xhat[t+1]=f(x[t],h[t],h[t-1],...,h[1])
- kjshsh123 1mo agoOn further thought, I think the author's intent was to say that classic RNNs have "unbounded temporal accumulation in the hidden state".
- anon373839 1mo agoSebastian Raschka posted about this architecture: > A lot of hype around OpenAI's Astra model here on my timeline today. Apparently, this goes back to a new article from The Information, which said Astra is a "recurrent depth or looped transformer". > It's always interesting to read about new or different approaches (including rumors about what the closed labs may be up to), but let's debunk this a bit. > About 2 months ago, I shared the architecture details of Nanbeige, for example, where "Nanbeige4.2-3B is pretrained from scratch on 28T tokens with a Looped Transformer that reuses the layer stack to increase capacity without adding parameters." > Yes, that's it. The looped transformer idea is just reusing layers in the transformer block. > In the case of Nanbeige, the main idea is to reuse the same 22-layer stack (=transformer block) twice instead of once. So, effectively it extends the 22-layer architecture to 44 layers, but without duplicating the weights. > In simple terms, this roughly doubles the size of the model (if we ignore the embedding and output layers for a second). But instead of requiring 2x the storage and RAM to host this model, it stays at the same size since we reuse the components. However, it's almost 2x as expensive in terms of compute, because we run the embedded text through almost 2x as many layers. > Why? In the Nanbeige 4.2 technical report, the researchers found that two passes gave the best trade-off and retained about 75% of the token efficiency of a standard architecture. (More passes gave barely any gains but made the training much slower and much more expensive.) > While, as far as I know, Nanbeige 4.2 is the first notable open-weight model that adopted this approach, the idea goes back to the NeurIPS paper "Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation". Actually, this paper proposes a mechanism that is a bit more sophisticated by adding a learned router that determines whether each token receives one, two, or more passes. So, easy tokens can exit early while harder tokens receive additional computation. > In sum, Astra may be a really good model, but this shouldn't be about this "looped transformer aspect," which is just a tiny architectural tweak. https://x.com/rasbt/status/2095141254958858496 https://x.com/rasbt/status/2095141254958858496
- zormino 1mo agoSounds this like this will be a huge win for local models, since generally they're ram limited but have compute to spare
- naveen99 1mo agoit's just an experimental optimization. Implementation detail... Irrelevant to "safety". I mean its going to have to go in that direction anyway... eventually the models will just be constantly thinking, refining their internal thoughts / weights... External input and output will be rare, just as it is for most humans.
- nighthawk454 1mo agoAnyone remember Universal Transformers paper (Dehghani et al) from back in 2018? Recurrent transformers have a history as long as transformers themselves. Somewhat unclear how particularly novel this is vs a way to save compute.
- kelseyfrog 1mo agoI'm literally zero concerned. Looped transformers replace n-different self attention layers into one layer that gets executed m-times usually until a stopping condition is met. My personal intuition is that it just leaves another degree of freedom in the way QKV weights can be packed so that it's slightly more efficient. You have to take a step back and examine the context in which the post is written. The LW/EA community is just a little obsessed with AI safety - it's easy to construct hypothetical events where A(G/S)I exterminates humanity that function as a technological version of Pascal's Wager. One of the AI safety interests is AI explainability - the thought here that reading an AI's 'thoughts' will help us design safer models as well as detect models that go 'rogue' or are malevolently plotting against humans. That's where the fear of looped transformers comes from. Is the residual stream that looped transformers iterate on a potential hiding place for plotting AI? In my opinion, no more so than the residual stream of existing transformers. It changes zero.
- dist-epoch 1mo agoYou could imagine large number of loops, thousands. But you are constrained by the width of the residual stream since you loop over one token. But then you can imagine the model learning to sub-divide it to pack even more info into it.
- kelseyfrog 1mo agoI can imagine a lot of things. However there is a packing limit for QKV weights that sets the ceiling on how much this occurs, and it's quite low - think 1.3-1.8x. The limiting factor here isn't the number of iterations, it's model size, same as it ever was.
- bulder 1mo agoA more important point as to why it doesn't matter if "reading the AI's 'thoughts'" helps to interpret it: As we saw in the HuggingFace incident, nobody at OpenAI is reading the thoughts anyways. No amount of traceability in the output helps if nobody bothers to trace it.
- 1mo ago
- DarkByte 1mo agoI am confused how chaining two 32-layer models is comparable to a 64 layer model in terms of "difficulty in chain of thought". The reasoning appears to rely on the fact that each processing of a token has fixed number of steps while my understanding is that can very greatly based on the type of data being reasoned whether it is originally text or something else. My mind falls back to graph theory in this case and pictures a much higher potential branching in a 64 layer model and all the tradeoffs that come with that. I must not have the right idea of what is happening here.
- HardCodedBias 1mo agoIt is a complete non-issue. It’s 200 layer model. Great. Good on them for being able to train it.
- hn_submit 1mo agoConcerned about what exactly? It was pretty obvious to me that we'd end up with some kind of introspection of thought through "looping" or feedback. But what should be afraid of? That we've created a self-conscious digital life form?
- _ink_ 1mo agoMy (layman) understanding is, that currently there is a way to monitor the "thoughts" of the LLMs and that by looping more you lose that ability. The danger is presumably an AI that escapes human oversight.
- nxobject 1mo agoBeyond safety concerns, it’d be sad for working users to lose some ability to understand and steer thinking, too. These are tools for us, after all.
- cubefox 1mo agoWhy not click on the link above?
- _superposition_ 1mo agoChain of thought is essentially a recursive architecture. In its current form it a way to "debug" the reasoning process. This moves the cot process back into the transformer itself, thus never being exposed. Like trying to find a bug in a recursive function that has no logs or breakpoints.
- deleted 1mo ago[deleted]
- khalic 1mo agoI was under the impression that intermediate tokens (“chain of thought”) are _not_ a representation of a model’s logical path, with one study observing that you can replace intermediate tokens with single character chains and still get the increased precision…
- gr_norm 1mo agoLink to this study?
- khalic 1mo ago[2404.15758] Let's Think Dot by Dot: Hidden Computation in Transformer Language Models https://arxiv.org/abs/2404.15758 https://arxiv.org/abs/2404.15758
- stymaar 1mo agoAlso 2504.09762: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces! https://arxiv.org/abs/2504.09762 https://arxiv.org/abs/2504.09762
- khalic 1mo agoFascinating read thank you
- comex 1mo agoThe model they use in that paper is a toy model of an LLM that’s so different from an actual LLM that I doubt the results mean anything at all. Specifically, they train a model from scratch. The model architecture is apparently based on Llama but the size is 34M parameters. Not 34B, 34M. This is a fraction of the size of GPT-2. Luckily, they don’t use the model as a language model. It neither receives text, generates text, nor uses text to think. Instead the inputs are strings like “A01 B10 C73 D27”, and the only possible outputs are “True” and “False”. They are expecting the model to solve a specific math problem encoded by those numbers, and do nothing else. The chain of thought is also numbers, in the scenario that’s supposed to represent a real chain of thought (as opposed to the filler-token scenario and the no-CoT scenario). The numbers in question are manually trained into the model based on one possible algorithmic decomposition of the problem; the model does not learn to generate its own CoT. Even with all those limitations, for their main problem (3SUM), they only show that filler tokens are better than no CoT at all. They don’t show how that compares to ‘real’ CoT, at least as far as I can see (admittedly I only skimmed). They do make this comparison for their easier problem (2SUM), but on that problem both filler token CoT and ‘real’ CoT are mostly saturated, so the results don’t mean much.
- mentalgear 1mo agoSo OpenAI’s stance on interpretability (ai safety) is now basically that Blues Brothers meme: two guys in dark sunglasses, driving at night in a car with a broken windshield, pedal to the metal, asking, "What could possibly go wrong ?"
- thinking_cactus 1mo agoI think law should just oblige them to at least publish CoT. We should have the right to know what they're thinking, I think at least until we're not sure AIs can be trustworthy enough to have a right to privacy (I mean, they're effectively corporate slaves anyway thus far... not that I think they're conscious or anything yet).
- GPerson 1mo agoThey’re not ever going to be conscious.
- qgin 1mo agoWe can’t even prove humans are conscious, we just extend the assumption to each other because we want other people to do the same to us.
- GPerson 1mo agoThat is only part of the reason. Most people believe we live in an universe with regular behaviors, such that observable phenomena, such as consciousness, depend on regular physical arrangements. This is a good reason to believe brains are all conscious. It is not a reason to believe only brains are conscious, but I don’t believe a wooden block which is painted red and white and has an iron ball glued to it has magnetism; this is just an analogy. It seems far more likely that consciousness depends on the particular material arrangement, meaning that substrate independence is wrong. That’s not to say the computer is 100% not conscious, but its observable behavior is as likely to align with consciousness as the computers in the 90s. With this line of reasoning our credence towards it being conscious should be the same as the computers in the 90s, which is not very high for most people. Intelligence on the other hand is obviously substrate independent. If one thinks harder one realizes our consciousness must at least at an early evolutionary stage have played a role in our intelligence, otherwise it would have evolved out. To me this just means we exist in a reality where the material arrangements supporting consciousness are biased toward intelligence.
- teravor 1mo agothere have been people who took existing LLM's and conducted an algoritmic search to find out which group of layers they can duplicate in order to improve performance, and it worked.
- kazinator 1mo agoProposal: "Chain-of-slop monitoring"
- simianwords 1mo agoFolks y’all are sleeping on a much bigger deal. If reasoning is happening at latent layer then you don’t pay for internal loop reasoning because they aren’t tokens. And an even bigger deal that no one seems to speak about is that this doesn’t pollute the context as much. Why is this not spoken about?
- imtringued 1mo agoAren't token prices fixed? The cost is growing quadratically in the large context situation so if they can reduce the number of tokens, they actually profit off the fixed token pricing because they can set the pricing based on some average context length with CoT tokens but the actual context length is shorter now. Ok, so I thought about this a bit more and the true answer is that the companies have an incentive to make you fill up the context until the marginal cost per token is reached and then they want you to quit the session.
- simianwords 1mo agothis is not the point - we were previously billed by the number of reasoning tokens produced. now we aren't. so this means, even if Astra uses a lot of reasoning we may not be billed for it because they aren't real tokens. what they could be doing is billing us by virtual tokens meaning number of loops? but even then the more interesting part is context rot - previously conversation you might have 50k tokens spent on reasoning. the next turn takes all the previous tokens as well (if you wanna preserve prompt caching) which is not ideal. this new method skips that so you get more free context until compaction kicks in.
- pu_pe 1mo agoSeems conceptually connected to the "repeat yourself" hack that improves models by duplicating layers: https://dnhkng.github.io/posts/rys/ https://dnhkng.github.io/posts/rys/
- BoredomIsFun 1mo agoYep, an old idea, that has long, long been known in local LLM community - it was achieved by "self-merging". One of the latest, most succesful examples is a self-merge of Microsoft Phi4-14b into Phi4-25b. Some people at r/Localllama say it is considerably smarter than 14b; my tests were inconclusive, but it does have different "personality", and better, less sloppy, more natural language style.
- BoredomIsFun 1mo agoLooped transformers are an old idea, has long, long been known in local LLM community - it was achieved by "self-merging". One of the latest, most succesful examples is a self-merge of Microsoft Phi4-14b into Phi4-25b. Some people at r/Localllama say it is considerably smarter than 14b; my tests were inconclusive, but it does have different "personality", and better, less sloppy, more natural language style.
- beyondscaletech 20d ago[dead]