8 ms·
Training LLMs to Reason in a Continuous Latent Space
- ilaksh 2y agoIt seems like the latent space could be even more useful if it was trained with the transcribed videos.
- mentalgear 2y ago"We utilize the last hidden state of the LLM as a representation of the reasoning state (termed "continuous thought")." Could someone explain the last hidden state of the LLM ? What it shape is and how it is normally used - and why it hasn't been used yet to augment the next input? (which seems logical)
- AmazingTurtle 2y agoEmbeddings aka the last hidden state are the mathematical representation of an input of the model before a separate model (usually the decoder) translates that hidden state to a next token (the generative part in generative ai). Normally, the this step repeats over and over. This novel approach introduces re-using the last hidden state as if it was a token that has been generated thus "evolving" the hidden state over each iteration.
- tjbai 2y agoThe last hidden state is just the output embedding after N residual layers, e.g. input embedding + res1 + res2 + ... There's typically an "unembedding layer"/"classification head" that uses this hidden state to produce a softmax distribution over the LLM's vocabulary. In this case, we can think of this as "snapping" the hidden state into a single token and feeding that token into the next position of the autoregressive LLM. In this sense, the last hidden state _does_ augment the next input. The authors simply propose directly feeding this hidden state into the next step rather than reducing it into a single token—thus, reasoning in continuous latent space rather than discrete token space.
- intalentive 2y agoMoreover “snapping” the hidden state to a token is akin to quantization. It’s lossy. By staying in latent space the model can “reason” at “full resolution” without discretization noise.
- soulofmischief 2y agoUseful information about conceptual relationships and procedure can be captured in the LM head, so there is also potential lossiness when short-circuiting it.
- snthpy 2y agoSometimes discretization introduces interesting behavior though. Compare for example the logistic map and it's chaotic regime with the simplicity of the logistic ODE. Another example would be quantum mechanics compared to classical mechanics and determinism. The Poincare Conjecture was only interesting for n=3 due to too much connectivity in higher dimensions. Wouldn't it be interesting if consciousness only arose in such a discretized form, a case of incidental complexity and chaos introduced as the result of topological non-triviality from quantization? Don't forget, non-linearity is fundamental to the whole process, otherwise you'd just have one large linear transformation. Maybe there's a similar role for discretization? :shrug:
- sweetheart 2y agoWow this was the explanation that made it all click for me. Thanks so much!
- psb217 2y agoThe way the recurrence in this method works -- ie, using last LLM hidden state at previous time step as input token for the next time step -- isn't directly compatible with how recurrence/autoregression is typically handled during LLM training. One of the major strengths of transformers is that they can be trained for recurrence/autoregression (which have sequential dependency) using convolutions (which are embarrasingly parallel). The proposed method requires introducing some sequential dependencies during training that could otherwise be avoided using "causal masking" and convolutions to enforce the correct dependencies between time steps in a sequence. Introducing these sequential dependencies makes training a lot slower. tldr; the method requires training in a way that loses one of the major benefits of transformers, but maybe in some scenarios that loss is worth it.
- DalasNoin 2y agoSo the models will no longer be thinking in plain English but some embedding space? Seems not like what you want.
- Vampiero 2y agoSeems exactly like what you want. We don't think in plain English, we _rationalize_ our thoughts into English (or whatever language comes out) but they must be more fundamental than language because language is acquired. Essentially, English is one of many possible encodings of an underlying intuitive, possibly non-symbolic representation.
- ekianjo 2y ago> We don't think in plain English That's debatable. Language shapes thoughts much more than you might think. Because you learn concepts from language that you could not imagine by yourself until you learned/read about them, so they are in effect very linked to language.
- Vampiero 2y agoFair, but there are many categories of languages. For example, I can think in formal logic. I've learned to do that, and surely my brain takes a step-by-step approach to it, but I've also internalized some of it and I don't think that my proficiency with English has anything to do with it. I could have learned the same concepts in any other language, but the end result would be the same. And surely there are many thoughts that can't be expressed purely with words. For example all that is related to qualia. You can think of a color but you can't describe what you see in your mind's eye with words, not in a way that would let a blind person share the same experience. Or try describing "love" without making a similitude. Is love a thought? Or a feeling? Is there a meaningful difference between the two?
- samiskin 2y agoLanguage is definitely a significant part of thinking, but when I remember how cold it was outside yesterday to figure out if it was colder than today, I'm not bringing words to mind. I'm bringing up some other non-discrete information that I could never precisely encode into words and then factoring that in with the other non-discrete information I'm currently taking in through my senses. Its only after that processing that I encode it as a lossy "It was colder yesterday" statement.
- vouaobrasil 2y ago> Experiments show that Coconut can effectively augment the LLM on several reasoning tasks. It really seems like we are building a true intelligence, adding components to different parts of a "brain" until we have something rivalling the human mind. It's exceptionally dangerous and it's remarkable how researchers turn a blind eye to any possible consequences.
- AmazingTurtle 2y agoOne day researchers will be like "Oh crap what have we done" and "Shut it down, shut it down!!!"
- vouaobrasil 2y agoThat is true. Most poeple will just respond to immediate physical threats as long as they have the illusory safety net of modern society.
- ionwake 2y agoJust bear in mind while they are yelling "shut it down" there will be a bunch of commenters with no idea whats happening saying that they are just over reacting
- ionwake 2y agoagree. Someone should make sure the next ASI develops an extension to hide the comments in every AI thread 80% full of the brightest minds saying " I tried to build a react app and it totally failed doing it the way I wanted ".
- rsrsrs86 2y agoIf ASI is artificial specific intelligence, I bet your pardon; intelligence can hardly be specific. Intelligence can reflect upon itself.
- 2y ago
- anon291 2y agoIt's a good, understandable paper. The main issue with chain-of-thought (which I think is a solid approach, and one that needs to take place) is that we ourselves aren't necessarily 'trained' on chain-of-thought. Yes, we do learn mathematical proofs and reasoning at some point (usually), but most people settle on latent thinking without training, and switch between the two modes naturally. My intuition says we're missing something, but who knows
- empath75 2y agoI wonder what would happen if you just ran this on a continuous loop and only intermittently fed in new tokens or queried it for token outputs.
- rsrsrs86 2y agoWell, if you consider the case of a linear regression, fitting on your output will add no new information to the weights. Try that on any notebook.
- SubiculumCode 2y agoIn LLMs, is there a correlation between layer depth and the activations correspondence to the abstract to concrete details continuum?
- pizza 2y agoYes: for eg BPE, due to how it progressively pushes compound tokens of already seen - hence more common - subtokens to the ‘top’ of the vocab), you can train a model to do regression over vocabulary index for the next token from the current token embedding - using the same single regression model for all layer depths. If you plot mse of token index prediction versus layer depth then you can see that the mse of the prediction decreases steadily per additional layer. This appears to be because token index in eg BPE is actually fairly smooth and so it seems like the model is capable of localizing to the actual correct vocab index as depth increases, so kind of like a fuzzy->discrete refinement as you go deeper in layers https://arxiv.org/abs/2408.13442 https://arxiv.org/abs/2408.13442
- SubiculumCode 2y agoTHANKS!
- fabmilo 2y agoI like the direction of the research of working in latent space but feeding the last layer representation back as a first layer embedding feels sketchy to me. Those layers have different representation space.
- zxexz 2y agoFeeding the last layer back as the input embedding has been done many times, e.g. Transformer-XL. The models are trained like this, it's not like they're taking a pre-trained Llama and just feeding it to itself. It's a simple, computationally cheap mechanism to add feedback.
- empath75 2y agoI read a paper not long ago that showed that deleting, duplicating and reordering layers doesn't actually seem to matter that much and it feeding back is just a kind of re-ordering.
- TeMPOraL 2y agoSo you're saying that feeding the last layer back to the first makes the model layer-order independent, or kinda infinitely deep, if you squint? :).
- torginus 2y agoImo this kind of makes sense - LLMs without a feedback loop can learn to have one themselves by encoding information in the previously generated tokens.
- imtringued 2y agoThey can't, because that would increase training loss. The training loss acts as a gatekeeper for reasoning.
- fabmilo 2y agofrom my understanding that is what they do, see the paper: > We use a pre-trained GPT-2 (Radford et al., 2019) as the base model for all experiments. I agree the feedback is necessary, and the mechanism simple and cheap, but I don't think is optimal.
- TibbityFlanders 2y ago[dead]
- bick_nyers 2y agoI wonder if you would want to use an earlier layer as opposed to the penultimate layer, I would imagine that the LLM uses that layer to "prepare" for the final dimensionality reduction to clean the signal such that it scores well on the loss function.
- patcon 2y agoI think of an LLM model as like a crystallised mathematical snapshot of intelligence... like a cell on a microscope slide, a dead and mounted form of output from the living process of intelligence... This paper makes me wonder whether, in a very fuzzy sense, we could give #LLMs access to some similarly crystallised analog of emotion or emotional valence, below the level of language https://x.com/patcon_/status/1866549080127893613?s=46 https://x.com/patcon_/status/1866549080127893613?s=46
- threeseed 2y agoIntelligence is more than just knowing the probabilistic relationship between every word.
- Rhapso 2y ago"Intelligence" is a continuous process. Without a continuous feedback loop, LLMs will never be more than a compression algorithm we bullied into being a chatbot. OpenAi as a mega-organism might be intelligent, but the LLMs definitely are not. The "compressed capture of semantic relationships" is a new thing we don't have a word for.
- thrance 2y agoFunnily enough, there is a mathematical link between data compression and AGI [1]. I believe a paper circulated some time ago that compared gpt2 to gzip, with interesting results. [1] https://en.wikipedia.org/wiki/AIXI https://en.wikipedia.org/wiki/AIXI
- ttul 2y agoI've been looking into using the last hidden layer of an off-the-shelf LLM to help my company with a classification task. The last hidden layer is obviously super rich in semantic information because it has to somehow tell the next layer how to generate the next token prediction. That final layer, in some respects, is discarding valuable context information that the final hidden layer encodes. I am not surprised at all that Meta was able to generate some positive returns by feeding the last hidden layer back into the model auto-regressively. The method of training they describe in the paper is really cool. Summarized in Figure 2, they train it with a corpus of step-by-step text instructions and then across multiple stages, they iteratively replace one of the textual steps with a last-hidden-layer embedding and see what the model spits out. The weights are then updated through cross-entropy loss as the additional text tokens are generated once again. So they're basically rewinding the output, replacing an increasing number of textual steps with hidden state embeddings, and playing it forward as the model gradually learns to do all of its step-by-step thinking using just the hidden state data. In a way, this might be how humans learn to think through language. Our parents teach us using words and our brain gradually replaces the words with thoughts until we can replicate the action or solve the problem ourselves without anyone guiding us with words.
- ttul 2y agoIndeed, I would not be surprised if OpenAI one day admits that the `o1` model uses the last hidden layer (or some other intermediate layer) to feed the "thought process" that you can watch as it "thinks" about the answer. I suspect that they may take the last hidden layer and feed it back into the front of the `o1` model while also feeding a separate, likely much smaller LLM that generates the "thought process" as language tokens. In this manner, the model makes use of the rich semantic information encoded at the last hidden layer while informing the user via an extraction of that hidden layer specifically tuned to generate human-legible concepts such as, "I'm considering the impact of converting the units from kilograms to pounds," or whatever.
- pedrovhb 2y agoThat's certainly possible, but it reminds me a bit of a similar thing I've seen in their UI that rhymes in a way that makes me think otherwise. In the code interpreter tool, you have a little preview of the "steps" it's following as it writes code. This turns out to just be the contents of the last written/streamed comment line. It's a neat UI idea I think - pretty simple and works well. I wouldn't be surprised if that's what's going on with o1 too - the thought process is structured in some way, and they take the headings or section names and just display that.
- thoughtlede 2y agoPerhaps these findings might be indicating that we need more NN layers/attention blocks for performing reasoning. This project circumvented the lack of more trained layers by looping the input through currently trained layers more than once. Also we may have to look for better loss functions than ones that help us predict the next token to train the models if the objective is reasoning.
- rsrsrs86 2y agoI feel abdominal pain when I see the words “thinking” or “reasoning” related to LLMs. I feel back pain when I read the crazy, unsound speculation about how the brain is supposed to be like a computer. Serious mistake.
- js8 2y agoI agree with you. "Chain of thought" is not reasoning, just like LSD trip isn't. I think we lack a good formal definition of what (fuzzy) reasoning is. Without it, we will always have some kind of unexplained hallucinations. I also believe AGI could be implemented as a model that can train models for specific tasks completely autonomously. But that would kill the cash cow, so OpenAI etc. are not interested in developing it.
- 234120987654 2y ago100% agree. I miss the days where the title would describe the method instead of being a sales pitch
- vidarh 2y agoUnless you can show an example of humans reasoning solving a problem outside the Turing computable set, there is no rational basis for assuming the brain is anything but a computer, as the very notion that we exceed Turing computability would be revolutionary and utterly mindbending in terms of consequences on a number of fields.
- platz 2y agothere is no rational basis for assuming the brain is a "computer" in the same way an intel x86 chip is a "computer" or that the universe is a "computer". Using language in this way without defining terms like what even is a computer is folly.
- vidarh 2y agoThere is no rational basis for assuming it is not, as we have not a single example of a computable function outside the Turing computable set. The term "computer" has it's original outside of "electronic computer". It used to be a role, a job function. There has been no time in human history where the only computers have been electronic computers. But, sure, let's be more precise: Any Turing complete system is equivalent to any Turing complete computer and can reasonably be called a computer, but let's also limit it to any system that can not compute functions outside the Turing computable set. We don't know of any such systems that have been shown to compute functions outside the Turing computable set, at all, including brains. The rational basis for assuming the brain is a computer is that we have not a single shred of evidence that exceeding Turing computability is possible, nor any theory for how to even express a function that is computable for humans but not Turing computable. If you can find one single such example, there'd be a rational basis for saying the brain isn't a computer. As it stands now, assuming it isn't, is nothing more than blind faith. .
- max93 2y agoWe conducted similar research earlier and successfully improved performance to a level comparable to models with 3x larger layer sizes. https://arxiv.org/html/2409.14199v3 https://arxiv.org/html/2409.14199v3 We utilize more computational time in the latent space to achieve better performance. However, this approach introduces greater resistance compared to Chain of Thought (CoT) reasoning in the token space, especially if the number of CoT rounds in the latent space exceeds 20. I would using the term "better approximation of the data distribution" instead of "reasoning" to describe this kind of process.
- cootsnuck 2y agoSo perhaps could be useful for fine-tuning on smaller models?
- max93 2y agoI think so. I believe this type of reasoning method, which achieves better results through longer computation time, is very useful on edge devices like mobile phones. Consider a scenario where we only need the model to output a function/action call on the phone; we don't require it to provide an immediate response.
- padolsey 2y agoWas it just me who thought that this was _already_ how LLMs worked? I'd always assumed they were -- so to speak -- swimming in their own embeddings space before coming out on the other side with language. But it turns out they're just feeding their own incremental outputs back into themselves, without a memory of the path they took to get there. Yowzer!
- vrighter 2y agoThere isn't any memory of how it got to where it did because all weights are evaluated all the time. It got there through the entirety of the network. There is no logic, just (mostly) a bunch of multiply-accumulates.
- deleted 2y ago[deleted]