13 ms·
Visualizing Attention, a Transformer's Heart [video]
- jacksonhacker 3y ago[dead]
- shahbazac 3y agoIs there a reference which describes how the current architecture evolved? Perhaps from very simple core idea to the famous “all you need paper?” Otherwise it feels like lots of machinery created out of nowhere. Lots of calculations and very little intuition. Jeremy Howard made a comment on Twitter that he had seen various versions of this idea come up again and again - implying that this was a natural idea. I would love to see examples of where else this has come up so I can build an intuitive understanding.
- ollin 3y agokarpathy gave a good high-level history of the transformer architecture in this Stanford lecture https://youtu.be/XfpMkf4rD6E?si=MDICNzZ_Mq9uzRo9&t=618 https://youtu.be/XfpMkf4rD6E?si=MDICNzZ_Mq9uzRo9&t=618
- HarHarVeryFunny 3y agoRoughly: 1) The initial seq-2-seq approach was using LSTMs - one to encode the input sequence, and one to decode the output sequence. It's amazing that this worked at all - encode a variable length sentence into a fixed size vector, then decode it back into another sequence, usually of different length (e.g. translate from one language to another). 2) There are two weaknesses of this RNN/LSTM approach - the fixed size representation, and the corresponding lack of ability to determine which parts of the input sequence to use when generating specific parts of the output sequence. These deficiencies were addressed by Bahdanau et al in an architecture that combined encoder-decoder RNNs with an attention mechanism ("Bahdanau attention") that looked at each past state of the RNN, not just the final one. 3) RNNs are inefficient to train, so Jakob Uszkoreit was motivated to come up with an approach that better utilized available massively parallel hardware, and noted that language is as much hierarchical as sequential, suggesting a layered architecture where at each layer the tokens of the sub-sequence would be processed in parallel, while retaining a Bahdanau-type attention mechanism where these tokens would attend to each other ("self-attention") to predict the next layer of the hierarchy. Apparently in initial implementation the idea worked, but not better than other contemporary approaches (incl. convolution), but then another team member, Noam Shazeer, took the idea and developed it, coming up with an architecture (which I've never seen described) that worked much better, which was then experimentally ablated to remove unnecessary components, resulting in the original transformer. I'm not sure who came up with the specific key-based form of attention in this final architecture. 4) The original transformer, as described in the "attention is all you need paper", still had a separate encoder and decoder, copying earlier RNN based approaches, and this was used in some early models such as Google's BERT, but this is unnecessary for language models, and OpenAI's GPT just used the decoder component, which is what everyone uses today. With this decoder-only transformer architecture the input sentence is input into the bottom layer of the transformer, and transformed one step at a time as it passes through each subsequent layer, before emerging at the top. The input sequence has an end-of-sequence token appended to it, which is what gets transformed into the next-token (last token) of the output sequence.
- krat0sprakhar 3y agoThank you for this summary! Very well explained. Any tips on what resources you use to keep updated on this field?
- HarHarVeryFunny 3y agoThanks. Mostly just Twitter, following all the companies & researchers for any new announcements, then reading any interesting papers mentioned/linked. I also subscribe to YouTube channels like Dwarkesh Patel (interviewer) and Yannic Kilcher (AI News), and search out YouTube interviews with the principles. Of course I also read any AI news here on HN, and sometimes there may be interesting information in the comments. There's a summary of social media AI news here, that sometimes surfaces something interesting. https://buttondown.email/ainews/archive/ https://buttondown.email/ainews/archive/
- kordlessagain 3y agoWhat I'm now wondering about is how intuition to connect completely separate ideas works in humans. I will have very strong intuition something is true, but very little way to show it directly. Of course my feedback on that may be biased, but it does seem some people have "better" intuition than others.
- cs702 3y agoFantastic work by Grant Sanderson, as usual. Attention has won.[a] It deserves to be more widely understood. --- [a] Nothing has outperformed attention so far, not even Mamba: https://arxiv.org/abs/2402.01032 https://arxiv.org/abs/2402.01032
- stillsut 3y agoIn training we learn a.) the embeddings and b.) the KQ/MLP-weights. How well do Transformers perform given learned embeddings but only randomly initialized decoder weights? Do they produce word soup of related concepts? Anything syntactically coherent? Once a well trained high dimensional representation of tokens are established. can they learn KQ/MLP weights significantly faster?
- kjhenner 3y agoThe first time I really dug into transformers (back in the BERT days) I was working on a MS thesis involving link prediction in a graph of citations among academic documents. So I had graphs on the brain. I have a spatial intuition for transformers as a sort of analog to a message passing network over a "leaky graph" in an embedding space. If each token is a node, its key vector sets the position of an outlet pipe that it spews value to diffuse out into the embedding space, while the query vector sets the position of an input pipe that sucks up value other tokens have pumped out into the same space. Then we repeat over multiple attention layers, meaning we have these higher order semantic flows through the space. Seems to make a lot of sense to me, but I don't think I've seen this analogy anywhere else. I'm curious if anybody else thinks of transformers in this way. (Or wants to explain how wrong/insane I am?)
- promiseofbeans 3y agoHis previous post 'But what is a GPT?' is also really good: https://www.3blue1brown.com/lessons/gpt https://www.3blue1brown.com/lessons/gpt
- nostrebored 3y agoWorking in a closely related space and this instantly became part of my team's onboarding docs. Worth noting that a lot of the visualization code is available in Github. https://github.com/3b1b/videos/tree/master/_2024/transformers https://github.com/3b1b/videos/tree/master/_2024/transformer...
- sthatipamala 3y agoSounds interesting; what else is part of those onboarding docs?
- jiggawatts 3y agoIt always blows my mind that Grant Sanderson can explain complex topics in such a clear, understandable way. I've seen several tutorials, visualisations, and blogs explaining Transformers, but I didn't fully understand them until this video.
- chrishare 3y agoHis content and impact is phenomenal
- deleted 3y ago[deleted]
- bilsbie 3y agoI finally understand this! Why did every other video make it so confusing!
- thinkingtoilet 3y agoGrant has a gift of explaining complicated things very clearly. There's a good reason his channel is so popular.
- chrishare 3y agoIt is confusing, 3b1b is just that good.
- visarga 3y agoAt the same time it feels extremely simple attention(Q,K,V) = softmax (Q K^T √ dK ) @ V is just half a row; the multi-head, masking and positional stuff just toppings we have many basic algorithms in CS that are more involved, it's amazing we get language understanding from such simple math
- bilsbie 3y agoFor me I never had too much trouble understanding the algorithm. But this is the first time I can see why it works.
- diedyesterday 3y agoDo not be fooled by the simplicity; The magic itself is in the many Q, K and V matrices (each of which is huge) which are learned and depend on the language(s). This is just the form of the application of those matrices/transformations: Making the embedding for the last token of a context "attend to" (hence attention) all information (at all layers of meaning and not just syntactic or semantic meaning but logical, scientific, poetic, discoursal, etc. => multi-head attention) contained in the context so far. Any complex function can be made to look simple in some representation (e.g its Fourier series or Taylor series, etc.).
- 3y ago
- YossarianFrPrez 3y agoThis video (with a slightly different title on YouTube) helped me realize that the attention mechanism isn't exactly a specific function so much as it is a meta-function. If I understand it correctly, Attention + learned weights effectively enables a Transformer to learn a semi-arbitrary function, one which involves a matching mechanism (i.e., the scaled dot-product.)
- hackinthebochs 3y agoIndeed. The power of attention is that it searches the space of functions and surfaces the best function given the constraints. This is why I think linear attention will never come close to the ability of standard attention, the quadratic term is a necessary feature of searching over all pairs of inputs and outputs.
- mastazi 3y agoThat example with the "was" token at the end of a murder novel is genius (at 3:58 - 4:28 in the video) really easy for a non technical person to understand.
- hamburga 3y agoI think Ilya gets credit for that example — I’ve heard him use it in his interview with Jensen Huang.
- spacecadet 3y agoFun video. Much of my "art" lately has been dissecting models, injecting or altering attention, and creating animated visualizations of their inner workings. Some really fun shit.
- deleted 3y ago[deleted]
- j_bum 3y agoLink? Sounds fun and reminds me of this tweet [0] [0] https://x.com/jaschasd/status/1756930242965606582 https://x.com/jaschasd/status/1756930242965606582
- spacecadet 3y agoNah someone down voted it. And yes, it looks like that + 20 others that are animated.
- CamperBob2 3y agoDownvotes == empty boats. If "Empty Boat parable" doesn't ring a bell, Google it...
- globalnode 3y agounless an algorithm decides to block or devalue the content, but yeah i looked it up, very interesting parable, thanks for sharing.
- deleted 3y ago[deleted]
- spacecadet 3y agoanger is a gift
- thomasahle 3y agoI like the way he uses a low-rank decomposition of the Value matrix instead of Value+Output matrices. Much more intuitive!
- imjonse 3y agoIt is the first time I hear about the Value matrix being low rank, so for me this was the confusing part. Codebases I have seen also have value + output matrixes so it is clearer that Q,K,V are similar sizes and there's a separate projection matrix that adapts to the dimensions of the next network layer. UPDATE: He mentions this in the last sections of the video.
- namelosw 3y agoYou might also want to check out other 3b1b videos on neural networks since there are sort of progressions between each video https://www.3blue1brown.com/topics/neural-networks https://www.3blue1brown.com/topics/neural-networks
- rollinDyno 3y agoHold on, every predicted token is only a function of the previous token? I must have something wrong. This would mean that within the embedding of "was", which is of length 12,228 in this example. Is it really possible that this space is so rich as to have a single point in it encapsulate a whole novel?
- evolvingstuff 3y agoYou are correct, that is an error in an otherwise great video. The k+1 token is not merely a function of the kth vector, but rather all prior vectors (combined using attention). There is nothing "special" about the kth vector.
- causal 3y agoI read this comment yesterday and keep thinking about it. That final token really must "comprehend" everything leading up to it, right? In which case longer context lengths are just trying to pack more meaning into that embedding state. Which means the embedding model must do a lot of the lifting to be able to accurately represent meaning across long contexts so well. Now I want to know more about how those models are derived.
- diedyesterday 2y ago> "Is it really possible that this space is so rich as to have a single point in it encapsulate a whole novel?" Not with this GPT. The context size would not allow keeping attention to the total meaning of more than 2048 tokens (as reflected in the transformed embedding of that context's last token). For a substantial part of a novel, it would require a much larger context size with then presumably will need a higher dimensional embedding/semantic space.
- vanjajaja1 3y agoat that point what it has is not a representation of the input, its a representation of what the next output could be. ie. its a lossy process and you can't extract what came in the past, only the details relevant to next word prediction (is my understanding)
- tylerneylon 3y agoAwesome video. This helps to show how the Q*K matrix multiplication is a bottleneck, because if you have sequence (context window) length S, then you need to store an SxS size matrix (the result of all queries times all keys) in memory. One great way to improve on this bottleneck is a new-ish idea called Ring Attention. This is a good article explaining it: https://learnandburn.ai/p/how-to-build-a-10m-token-context https://learnandburn.ai/p/how-to-build-a-10m-token-context (I edited that article.)
- rahimnathwani 3y agoHe lists Ring Attention and half a dozen other techniques, but they're not within the scope of this video: https://youtu.be/eMlx5fFNoYc?t=784 https://youtu.be/eMlx5fFNoYc?t=784
- deleted 3y ago[deleted]
- danielhanchen 3y agoOh with Flash Attention, you never have to construct the (S, S) matrix ever (also in article) Since its softmax(Q @ K^T / sqrt(d)) @ V, you can form the final output in tiles. In Unsloth, memory usage scales linearly (not quadratically) due to Flash Attention (+ you get 2x faster finetuning, 80% less VRAM use + 2x faster inference). Still O(N^2) FLOPs though. On that note, on long contexts, Unsloth's latest release fits 4x longer contexts than HF+FA2 with +1.9% overhead. So 228K context on H100.
- mehulashah 3y agoThis is one of the best explanations that I’ve seen on the topic. I wish there was more work, however, not on how Transfomers work, but why they work. We are still figuring it out, but I feel that the exploration is not at all systematic.
- seydor 3y agoI have found the youtube videos by CodeEmporium to be simpler to follow https://www.youtube.com/watch?v=Nw_PJdmydZY https://www.youtube.com/watch?v=Nw_PJdmydZY Transformer is hard to describe with analogies, and TBF there is no good explanation why it works, so it may be better to just present the mechanism, "leaving the interpretation to the viewer". Also, it's simpler to describe dot products as vectors projecting on one another
- blt 3y agoAs a computer scientist, the "differentiable hash table" interpretation worked for me. The AIAYN paper alludes to it by using the query/key/value names, but doesn't explicitly say the words "hash table". I guess some other paper introduced them?
- nerdponx 3y ago> TBF there is no good explanation why it works My mental justification for attention has always been that the output of the transformer is a sequence of new token vectors such that each individual output token vector incorporates contextual information from the surrounding input token vectors. I know it's incomplete, but it's better than nothing at all.
- rcarmo 3y agoYou're effectively steering the predictions based on adjacent vectors (and precursors from the prompt). That mental model works fine.
- eurekin 3y ago> TBF there is no good explanation why it works I thought the general consesus was: "transformers allow neural networks to have adaptive weights". As opposed to the previous architectures, were every edge connecting two neurons always has the same weight. EDIT: a good video, where it's actually explained better: https://youtu.be/OFS90-FX6pg?t=750&si=A_HrX1P3TEfFvLay https://youtu.be/OFS90-FX6pg?t=750&si=A_HrX1P3TEfFvLay
- mjburgess 3y ago
- abotsis 3y agoI think what made this so digestible for me were the animations. The timing, how they expand/contract and unfold while he’s speaking.. is all very well done.
- _delirium 3y agoThat is definitely one of the things he does better than most. He actually wrote his own custom animation library for math animations: https://github.com/3b1b/manim https://github.com/3b1b/manim
- divan 3y agoAlso check out community edition: https://www.manim.community https://www.manim.community
- rayval 3y agoHere's a compelling visualization of the functioning of an LLM when processing a simple request: https://bbycroft.net/llm https://bbycroft.net/llm This complements the detailed description provided by 3blue1brown
- lying4fun 3y agoamazing visualisation
- bugthe0ry 3y agoWhen visualised this way, the scale of GPT-3 is insane. I can't imagine what 4 would like here.
- spi 3y agoIIRC, GPT-4 would actually be a bit _smaller_ to visualize than GPT3. Details are not public, but from the leaks GPT-4 (at least, some by-now old version of it) was a mixture of expert, with every model having around 110B parameters [1]. So, while the total number of parameters is bigger than GPT-3 (1800B vs. 175B), it is "just" 16 copies of a smaller (110B) parameters model. So if you wanted to visualize it in any meaningful way, the plot wouldn't grow bigger - or it would, if you included all different experts, but they are just copies of the same architecture with different parameters, which is not all that useful for visualization purposes. [1] https://medium.com/@daniellefranca96/gpt4-all-details-leaked-48fa20f9a4a https://medium.com/@daniellefranca96/gpt4-all-details-leaked...
- joaogui1 3y agoMixture of Experts is not just 16 copies of a network, it's a single network where for the feed forward layers the tokens are routed to different experts, but the attention layers are still shared. Also there are interesting choices around how the routing works and I believe the exact details of what OpenAI is doing are not public. In fact I believe someone making a visualization of that would dispell a ton of myths around what are MoEs and how they work
- justanotherjoe 3y agoIt seems he brushes over the positional encoding, which for me was the most puzzling part of transformers. The way I understood it, positional encoding is much like dates. Just like dates, there are repeating minutes, hours, days, months...etc. Each of these values has shorter 'wavelength' than the next. The values are then used to identify the position of each tokens. Like, 'oh, im seeing january 5th tokens. I'm january 4th. This means this is after me'. Of course the real pos.encoding is much smoother and doesn't have abrupt end like dates/times, but i think this was the original motivation for positional encodings.
- nerdponx 3y agoThat's one way to think about it. It's clever way to encode "position in sequence" as some kind of smooth signal that can be added to each input vector. You might appreciate this detailed explanation: https://towardsdatascience.com/master-positional-encoding-part-i-63c05d90a0c3 https://towardsdatascience.com/master-positional-encoding-pa... Incidentally, you can encode dates (e.g. day of week) in a model as sin(day of week) and cos(day of week) to ensure that "day 7" is mathematically adjacent to "day 1".
- deleted 3y ago[deleted]
- bjornsing 3y agoThis was the best explanation I’ve seen. I think it comes down to essentially two aspects: 1) he doesn’t try to hide complexity and 2) he explains what he thinks is the purpose of each computation. This really reduces the room for ambiguity that ruins so many other attempts to explain transformers.
- Xcelerate 3y agoAs someone with a background in quantum chemistry and some types of machine learning (but not neural networks so much) it was a bit striking while watching this video to see the parallels between the transformer model and quantum mechanics. In quantum mechanics, the state of your entire physical system is encoded as a very high dimensional normalized vector (i.e., a ray in a Hilbert space). The evolution of this vector through time is given by the time-translation operator for the system, which can loosely be thought of as a unitary matrix U (i.e., a probability preserving linear transformation) equal to exp(-iHt), where H is the Hamiltonian matrix of the system that captures its “energy dynamics”. From the video, the author states that the prediction of the next token in the sequence is determined by computing the next context-aware embedding vector from the last context-aware embedding vector alone. Our prediction is therefore the result of a linear state function applied to a high dimensional vector. This seems a lot to me like we have produced a Hamiltonian of our overall system (generated offline via the training data), then we reparameterize our particular subsystem (the context window) to put it into an appropriate basis congruent with the Hamiltonian of the system, then we apply a one step time translation, and finally transform the resulting vector back into its original basis. IDK, when your background involves research in a certain field, every problem looks like a nail for that particular hammer. Does anyone else see parallels here or is this a bit of a stretch?
- BoGoToTo 3y agoI've been thinking about his a bit lately. If time is non-continuous then could you model the time evolution of the universe as some operator recursively applied to the quantum state of the universe? If each application of the operator progresses the state of the universe by a single planck-time could we even observe a difference between that and a universe where time is continuous?
- BobbyTables2 3y agoI think Wolfram made news proposing something roughly along these lines. Either way, I find Planck time/energy to be a very spooky concept. https://wolframphysics.org/ https://wolframphysics.org/
- 3y ago