5 ms·

I've been looking into using the last hidden layer of an off-the-shelf LLM to help my company with a classification task. The last hidden layer is obviously sup

by ttul 2y ago

I've been looking into using the last hidden layer of an off-the-shelf LLM to help my company with a classification task. The last hidden layer is obviously super rich in semantic information because it has to somehow tell the next layer how to generate the next token prediction. That final layer, in some respects, is discarding valuable context information that the final hidden layer encodes. I am not surprised at all that Meta was able to generate some positive returns by feeding the last hidden layer back into the model auto-regressively. The method of training they describe in the paper is really cool. Summarized in Figure 2, they train it with a corpus of step-by-step text instructions and then across multiple stages, they iteratively replace one of the textual steps with a last-hidden-layer embedding and see what the model spits out. The weights are then updated through cross-entropy loss as the additional text tokens are generated once again. So they're basically rewinding the output, replacing an increasing number of textual steps with hidden state embeddings, and playing it forward as the model gradually learns to do all of its step-by-step thinking using just the hidden state data. In a way, this might be how humans learn to think through language. Our parents teach us using words and our brain gradually replaces the words with thoughts until we can replicate the action or solve the problem ourselves without anyone guiding us with words.

ttul 2y ago
Indeed, I would not be surprised if OpenAI one day admits that the `o1` model uses the last hidden layer (or some other intermediate layer) to feed the "thought process" that you can watch as it "thinks" about the answer. I suspect that they may take the last hidden layer and feed it back into the front of the `o1` model while also feeding a separate, likely much smaller LLM that generates the "thought process" as language tokens. In this manner, the model makes use of the rich semantic information encoded at the last hidden layer while informing the user via an extraction of that hidden layer specifically tuned to generate human-legible concepts such as, "I'm considering the impact of converting the units from kilograms to pounds," or whatever.
- pedrovhb 2y ago
  That's certainly possible, but it reminds me a bit of a similar thing I've seen in their UI that rhymes in a way that makes me think otherwise. In the code interpreter tool, you have a little preview of the "steps" it's following as it writes code. This turns out to just be the contents of the last written/streamed comment line. It's a neat UI idea I think - pretty simple and works well. I wouldn't be surprised if that's what's going on with o1 too - the thought process is structured in some way, and they take the headings or section names and just display that.
- impossiblefork 2y ago
  I don't think it does, because from this paper this kind of backfeeding is apparently quite difficult to train. I've said it before, but I think it's just something like Quiet-STaR, but simplified. They have a bunch of question answer pairs, many of which are difficult. They generate a lot of tokens from the question (let's say, 3x the length of the expected answer), summarise whatever is generated and reinforce whenever it generates the right answer. I don't think o1 is something complicated.
  - sigmoid10 2y ago
    o1 ist most likely just 4o optimized for CoT with some fine tuning or perhaps merely with a dedicated system prompt (which is probably the reason why they don't let you access it in the API) and enforced structured output. In fact you can recreate something very similar using 4o and the right system prompt + structured outputs.
psb217 2y ago
"...because it has to somehow tell the next layer how to generate the next token prediction." -- This isn't actually true in the case of transformers. Features in the final TF layer at time t in a sequence do not depend on the features in the final TF layer at any other time step. Recurrence in transformers is done "depthwise" via "causally masked" convolutions. Final layer features at time t can depend on penultimate layer features at time t-1, but not on final layer features at time t-1.
- danielmarkbruce 2y ago
  you are misunderstanding what the person is saying. They are saying the final hidden layer outputs a vector which has all the information that decides the logits which decide the probabilities of each token in the entire vocabulary. Ie, it is storing a lot of information.
  - ttul 2y ago
    Correct. And although the final layer outputs a softmax of the token probabilities, the model by that point surely has a rich understanding of more than just the next token it wants to predict.
    - danielmarkbruce 2y ago
      Yup, some tokens are effectively branching decisions. Yann has a whole rant about a shortcoming of LLMs being they take the same compute regardless of the position in a sentence - which isn't great because sometimes you really have a serious decision to make, other times not so much. It also makes you wonder about optimal embedding size - maybe the right size is 10x bigger.
    - versteegen 2y ago
      > surely has a rich understanding of more than just the next token it wants to predict > the last hidden layer is obviously super rich in semantic information I don't agree that this is obvious, and think it's likely wrong (see the sibling thread [1]). The model has to at some point compress down its prediction for the entire future string of text to a prediction for a single token. There's no prior reason to assume it does this mostly in the final "LM head" linear layer, and the inputs to it don't have to predict anything other than the very next token so there's no reason it should (which is what I think psb217 was getting at), but I'm not familiar with what research has been done into it. On the other hand, processing seems to typically be concentrated in the central layers. [1] https://news.ycombinator.com/item?id=42379167 https://news.ycombinator.com/item?id=42379167
WiSaGaN 2y ago
This is intriguing. When I learned that a lot of people do not have inner monologue, I was fascinated by the fact that people can differ on such seemingly fundamental way of being. Maybe those who have it just have a "tee" that pipes into words.
- bongodongobob 2y ago
  I'm not convinced they don't. Ask them what they do when they read. That's all an inner monologue is.
  - taylorius 2y ago
    That's true - though I think of an inner monologue as being more "self driven". Perhaps it's just that their mental voices don't spontaneously say anything.
  - deleted 2y ago
    [deleted]
  - Regic 2y ago
    Don't we all experience this from time to time? When I'm focused on solving some mathematical problems I'm not thinking in words, but in concepts. When you are thinking of words you also think of a concept, the only difference is that sometimes there are no words associated to it. Im my opinion, words, sentences are just a label to the thinking process, a translation of what is really going on inside, not the driver of it.
  - 542354234235 2y ago
    Not all people do have an inner monolog when reading, called subvocalization. Subvocalization is when you are basically reading to yourself inside your head, sounding out each word. It is one of the most common reasons for slow reading speed. Most people do not “need” to subvocalize and can train themselves to process the visual text directly, instead of first converting it to “auditory” information. I found this out a few years ago and I was shocked that the way I read wasn’t universal. I have since been practicing reducing/eliminating subvocalization and I am getting better, and it allows me to increase my reading speed significantly. It also serves as an excellent example of how different our internal mental processes can be, and how completely unaware we are that there could be any other way to think than our own.
throwawaymaths 2y ago
> using the last hidden layer iirc this is a well supported task iirc called "classification head" instead of "language modeling head" in case anyone else wants to do this as a fine-tuning project
liuliu 2y ago
BTW, people found that in-conext instruction is useful for these (for example, directly using the last hidden layer to condition a diffusion model is much worse than encoder-decoder model, but you can add instruction prefix "try to imagine more details with the following text: <prompt>" would enrich the last hidden layer vector to be superior than the encoder-decoder text features. Very interesting stuff.
- ttul 2y ago
  It’s so funny how you can basically tickle the model and it laughs.

by ttul 2y ago

ttul 2y ago

Indeed, I would not be surprised if OpenAI one day admits that the `o1` model uses the last hidden layer (or some other intermediate layer) to feed the "thought process" that you can watch as it "thinks" about the answer. I suspect that they may take the last hidden layer and feed it back into the front of the `o1` model while also feeding a separate, likely much smaller LLM that generates the "thought process" as language tokens. In this manner, the model makes use of the rich semantic information encoded at the last hidden layer while informing the user via an extraction of that hidden layer specifically tuned to generate human-legible concepts such as, "I'm considering the impact of converting the units from kilograms to pounds," or whatever.

pedrovhb 2y ago
That's certainly possible, but it reminds me a bit of a similar thing I've seen in their UI that rhymes in a way that makes me think otherwise. In the code interpreter tool, you have a little preview of the "steps" it's following as it writes code. This turns out to just be the contents of the last written/streamed comment line. It's a neat UI idea I think - pretty simple and works well. I wouldn't be surprised if that's what's going on with o1 too - the thought process is structured in some way, and they take the headings or section names and just display that.
impossiblefork 2y ago
I don't think it does, because from this paper this kind of backfeeding is apparently quite difficult to train. I've said it before, but I think it's just something like Quiet-STaR, but simplified. They have a bunch of question answer pairs, many of which are difficult. They generate a lot of tokens from the question (let's say, 3x the length of the expected answer), summarise whatever is generated and reinforce whenever it generates the right answer. I don't think o1 is something complicated.
- sigmoid10 2y ago
  o1 ist most likely just 4o optimized for CoT with some fine tuning or perhaps merely with a dedicated system prompt (which is probably the reason why they don't let you access it in the API) and enforced structured output. In fact you can recreate something very similar using 4o and the right system prompt + structured outputs.

psb217 2y ago

"...because it has to somehow tell the next layer how to generate the next token prediction." -- This isn't actually true in the case of transformers. Features in the final TF layer at time t in a sequence do not depend on the features in the final TF layer at any other time step. Recurrence in transformers is done "depthwise" via "causally masked" convolutions. Final layer features at time t can depend on penultimate layer features at time t-1, but not on final layer features at time t-1.

danielmarkbruce 2y ago
you are misunderstanding what the person is saying. They are saying the final hidden layer outputs a vector which has all the information that decides the logits which decide the probabilities of each token in the entire vocabulary. Ie, it is storing a lot of information.
- ttul 2y ago
  Correct. And although the final layer outputs a softmax of the token probabilities, the model by that point surely has a rich understanding of more than just the next token it wants to predict.
  - danielmarkbruce 2y ago
    Yup, some tokens are effectively branching decisions. Yann has a whole rant about a shortcoming of LLMs being they take the same compute regardless of the position in a sentence - which isn't great because sometimes you really have a serious decision to make, other times not so much. It also makes you wonder about optimal embedding size - maybe the right size is 10x bigger.
  - versteegen 2y ago
    > surely has a rich understanding of more than just the next token it wants to predict > the last hidden layer is obviously super rich in semantic information I don't agree that this is obvious, and think it's likely wrong (see the sibling thread [1]). The model has to at some point compress down its prediction for the entire future string of text to a prediction for a single token. There's no prior reason to assume it does this mostly in the final "LM head" linear layer, and the inputs to it don't have to predict anything other than the very next token so there's no reason it should (which is what I think psb217 was getting at), but I'm not familiar with what research has been done into it. On the other hand, processing seems to typically be concentrated in the central layers. [1] https://news.ycombinator.com/item?id=42379167 https://news.ycombinator.com/item?id=42379167

WiSaGaN 2y ago

This is intriguing. When I learned that a lot of people do not have inner monologue, I was fascinated by the fact that people can differ on such seemingly fundamental way of being. Maybe those who have it just have a "tee" that pipes into words.

bongodongobob 2y ago
I'm not convinced they don't. Ask them what they do when they read. That's all an inner monologue is.
- taylorius 2y ago
  That's true - though I think of an inner monologue as being more "self driven". Perhaps it's just that their mental voices don't spontaneously say anything.
- deleted 2y ago
  [deleted]
- Regic 2y ago
  Don't we all experience this from time to time? When I'm focused on solving some mathematical problems I'm not thinking in words, but in concepts. When you are thinking of words you also think of a concept, the only difference is that sometimes there are no words associated to it. Im my opinion, words, sentences are just a label to the thinking process, a translation of what is really going on inside, not the driver of it.
- 542354234235 2y ago
  Not all people do have an inner monolog when reading, called subvocalization. Subvocalization is when you are basically reading to yourself inside your head, sounding out each word. It is one of the most common reasons for slow reading speed. Most people do not “need” to subvocalize and can train themselves to process the visual text directly, instead of first converting it to “auditory” information. I found this out a few years ago and I was shocked that the way I read wasn’t universal. I have since been practicing reducing/eliminating subvocalization and I am getting better, and it allows me to increase my reading speed significantly. It also serves as an excellent example of how different our internal mental processes can be, and how completely unaware we are that there could be any other way to think than our own.

throwawaymaths 2y ago

> using the last hidden layer iirc this is a well supported task iirc called "classification head" instead of "language modeling head" in case anyone else wants to do this as a fine-tuning project

liuliu 2y ago

BTW, people found that in-conext instruction is useful for these (for example, directly using the last hidden layer to condition a diffusion model is much worse than encoder-decoder model, but you can add instruction prefix "try to imagine more details with the following text: <prompt>" would enrich the last hidden layer vector to be superior than the encoder-decoder text features. Very interesting stuff.

ttul 2y ago
It’s so funny how you can basically tickle the model and it laughs.