3 ms·
A Mathematical Framework for Transformer Circuits (2021)
- myself248 1mo agoNothing about B-H curves or magnetization current, must be that other kind of transformer, in that other kind of circuit.
- greenbit 1mo agoSomeone should step up and rectify that, there's a lot of potential for confusion.
- ziofill 1mo agoThis comment is hysteresical
- Bluestein 1mo agoI find it polarizing.-
- thatspartan 1mo agoIt can be quite eddyfying
- amelius 1mo ago> A previous project, the Distill Circuits thread, has attempted to reverse engineer vision models, but so far there hasn’t been a comparable project for transformers or language models. How successful was that?
- Bluestein 1mo agoThe entire thing went down: - https://distill.pub/2021/distill-hiatus/ https://distill.pub/2021/distill-hiatus/ ... and, here we are.-
- oofbey 1mo agoI’ve tried reading this many times. But it’s sooooo long. Is it worth it?
- Bluestein 1mo agoDepending on your field, degree of interest, general curiosity, its a must. Its up there with word2vec.-
- robrenaud 1mo agoNeal Nanda's walkthrough of it is pretty good. https://youtu.be/KV5gbOmHbjU?is=GSiv0rSHBocNdU8l https://youtu.be/KV5gbOmHbjU?is=GSiv0rSHBocNdU8l Also long, but it's detailed and complicated, so there is no escaping that.
- _jayhack_ 1mo agoShocking to me how little interest the general public has in mechinterp given the alien capabilities demonstrated by LLMs - this and the subsequent transformer-circuits.pub publications will be seen as classic, foundational work in a few years
- libraryofbabel 1mo agoThis is one of those papers that ought to have several textbook chapters unpacking some of the insights... I'll just pick a couple things I love: 1) There's a kind of "rabbit–duck illusion" moment where they show how you can reframe all the linear algebra around attention in a totally different way, demoting the Q, K, and V matrices in favor of emphasizing a set of much larger matrices that are mathematically equivalent and very useful for interpretability purposes. (So, don't trust anyone who insists that vague analogies about "keys" and "queries" are the only good way to understand attention! They probably haven't read this paper.) 2) They show the importance of the "residual stream" inside LLMs: it's not just a series of bypasses of layers that's useful for training stability (like I first had it explained to me) but a kind of main communication bus that runs unbroken through the whole model from embedding to output. Attention heads and Feedforward are off to the side, adding embeddings into the stream. Somehow I find this a much more satisfying way to understand LLM architectures, and whenever I see an architecture diagram now I mentally redraw it with the residual stream at the center.