18 ms·
Maybe it's clear to others, but it's certainly not to me, how exactly transformers - or rather transformer-based LLMs - are operating. I understand how transfo
by HarHarVeryFunny 3y ago
Maybe it's clear to others, but it's certainly not to me, how exactly transformers - or rather transformer-based LLMs - are operating.
I understand how transformers work, but my mental model is that a transformer is the processor and the LLM is an application that runs on it. After all, transformers can be trained to do lots of things, and what it learns when trained with a "predict next word" LLM objective is going to differ from what it learns (and hence operates?) in a different setting.
There have been various LLM interpretation papers analyzing aspects of them, such as the discovery of pairs of consecutive layer attention heads acting as "search and copy" "induction heads", and analysis of the linear layers as key-value stores, which perhaps leads to another weak abstraction of the linear layers as storing knowledge and perhaps the reasoning "program", with the "attention" layers being the mechanism being programmed to do the data tagging/shuffling ?
No doubt there's a lot more to be discovered about how these LLMs are operating - perhaps a wider variety of primitives built out of attention heads other than just induction heads ? It seems a bit early to be building a high level model of the primitives these LLMs have learnt, and not sure if attempting a crude transformer-level model really works given how the residual context is additive - it's not just tokens being moved around.
- deleted 3y ago[deleted]
- Juicyy 3y agoI saved this HN thread from a couple months ago that had a lot of great resources. https://news.ycombinator.com/item?id=35697627 https://news.ycombinator.com/item?id=35697627
- potatoman22 3y agoI think your mental model could be making LLMs seem more confusing than they are. LLMs are stacks of transformers and generative LLMs typically have another model that samples the transformer output. Maybe there's useful abstractions for analyzing them, but LLMs are just another deep learning model.
- HarHarVeryFunny 3y agoThe "attention" mechanism (a bit of a misnomer really) is what makes transformers more complex than many other neural nets - data isn't simply flowing through the model from layer to layer, but rather it is being copied and moved around by the attention heads. The "next word" it is generating doesn't even have to be a word it has ever seen before - it may be copying it from the prompt.
- potatoman22 3y agoInteresting. Any suggestions/references for learning about attention from this perspective?
- HarHarVeryFunny 3y agoThe paper I read was this one from Catherine Olsson et al at Anthropic. https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html https://transformer-circuits.pub/2022/in-context-learning-an... There's a useful article here that expands on the types of head composition and provides some illustrations. https://www.lesswrong.com/posts/TvrfY4c9eaGLeyDkE/induction-heads-illustrated https://www.lesswrong.com/posts/TvrfY4c9eaGLeyDkE/induction-...
- Legend2440 3y agoTransformers - and all other ML models - are ways to represent computer programs. You can think of them as a programming language designed to be easy for optimization instead of for human understanding. It's not clear to anybody exactly what kind of program structure LLMs have internally. Figuring that out is a major goal for the field of mechanistic interpretability.