3 ms·
Can you do that in the middle of the layers? And if you do, would that word be that meaningful to the final output? Genuinely curious.
by hangonhn 1y ago
Can you do that in the middle of the layers? And if you do, would that word be that meaningful to the final output? Genuinely curious.
- mbowcut2 1y agoYou can, and there has been some interesting work done with it. The technique is called LogitLens, and basically you pass intermediate embeddings through the LMHead to get logits corresponding to tokens. In this paper they use it to investigate whether LLMs have a language bias, i.e. does GPT "think" in English? https://arxiv.org/pdf/2408.10811 https://arxiv.org/pdf/2408.10811 One problem with this technique is that the model wasn't trained with intermediate layers being mapped to logits in the first place, so it's not clear why the LMHead should be able to map them to anything sensible. But alas, like everything in DL research, they threw science at the wall and a bit stuck.