3 ms·
On the one hand, this looks really useful. On the other hand: > There are various forms of attention / self-attention, Transformer (Vaswani et al., 2017) reli
by simonw 4y ago
On the one hand, this looks really useful.
On the other hand:
> There are various forms of attention / self-attention, Transformer (Vaswani et al., 2017) relies on the scaled dot-product attention: given a query matrix , a key matrix and a value matrix , the output is a weighted sum of the value vectors, where the weight assigned to each value slot is determined by the dot-product of the query with the corresponding key
There HAS to be a better way of communicating this stuff. I'm honestly not even sure where to start decoding and explaining that paragraph.
We really need someone with the explanatory skills of https://jvns.ca/ https://jvns.ca/ to start helping people understand this space.
- mirekrusin 4y agoAndrej Karpathy ie [0]? [0] https://www.youtube.com/watch?v=kCc8FmEb1nY https://www.youtube.com/watch?v=kCc8FmEb1nY
- benreesman 4y agoThe Illustrated Transformer is pretty great. I was pretty hazy after reading the paper back in 2017 and this resource helped a lot. https://jalammar.github.io/illustrated-transformer/ https://jalammar.github.io/illustrated-transformer/
- simonw 4y agoThanks, that one is really useful.
- eternalban 4y agoExcellent. Thanks for sharing.
- nerdponx 4y agoComplicated from whose perspective? I don't go around commenting on systems programming articles about how low-level memory management and currency algorithms are too complicated, or commenting on category theory articles that the terminology is too obtuse and monads are too hard. I agree that there probably could be a better "on ramp" into this material than "take an undergraduate linear algebra course", but ultimately it is a mathematical model and you're going to have to deal with the math at some point if you want to actually understand what's going on. Linear algebra and calculus are entry-level table stakes for understanding how machine learning works, and there's really no way around that.
- simonw 4y agoI am absolutely certain it is possible to explain this stuff without using jargon and mathematical notation that is impenetrable to the majority of professional software engineers.
- nerdponx 4y agoI agree, but that doesn't mean that every article on the subject needs to be written for that audience in that language.
- IanCal 4y agoAt some point, at some level, you really do need to just learn what a dot product is and what a matrix is. It's not weird notation or jargon, these are fundamental concepts. Just like if you want to really learn how programs work you can't refuse any explanation that talks about "variables" or "functions" because that's jargon. You can explain it, but it's going to be more at the level of "the network looks at the words" type explanation.
- mxkopy 4y agoThe idea of the transformer somehow being a trainable key-value store is kind of abstract and weird and has little to do with the mathematics of it. The math part of that is how the dot product encodes for similarity between vectors, but beyond that it really is a "if you get it you get it" kind of thing.
- peterfirefly 4y agoVector, matrix, weighted sum, and dot product are good places to start. In fact, these concepts are so useful that they are good places to start pretty much no matter where you want to go. 3D graphics, statistics, physics, ... and neural networks.
- maxutility 4y agoI found this article on “transformers from scratch”[0] to be a perfect (for me) middle ground between high level hand-wavy explanations and overly technical in-the-weeds academic or code treatments. [0] https://e2eml.school/transformers.html https://e2eml.school/transformers.html
- onos 4y agoPretty sure the current state is simply one of confused exploration … hopefully soon to be synthesized and simplified.
- TheDudeMan 4y agoAgree. Each new "advance" seems like just another trick/quirk (and adds jargon). Still waiting for the real advance.
- nmfisher 4y agoIt took me a long time to wrap my head around the whole "key/query/value" explanation (and to be honest I regularly forget which vector is which). I find the "weighted sum of vectors" explanation much simpler/more intuitive; this blog post is IMO the best on the subject: https://peterbloem.nl/blog/transformers https://peterbloem.nl/blog/transformers