7 ms·
It turns out that transformers have a learning mechanism similar to autodiff but better since it happens mostly within the single layers as opposed to over the
by adamnemecek 4y ago
It turns out that transformers have a learning mechanism similar to autodiff but better since it happens mostly within the single layers as opposed to over the whole graph. I wrote a paper on this recently https://arxiv.org/abs/2302.01834v1 https://arxiv.org/abs/2302.01834v1. The math is crazy.
- naasking 4y agoCan this all be done on the GPU so the CPU doesn't need to be involved to adjust the weights?
- LukeB42 4y agoCan you explain like I'm 5 why this matters distinctly from how transformers are normally trained with autodiff and what its possible applications are?
- adamnemecek 4y agoI’m talking about attention only transformers. Those don’t have an autodiff but still learn. The math is actually really cool.
- lostmsu 4y ago> attention only transformers Can you share any good link on the subject?
- adamnemecek 4y agohttps://transformer-circuits.pub/2021/framework/index.html https://transformer-circuits.pub/2021/framework/index.html
- lostmsu 4y agoMaybe I am missing something, but I don't see any learning without autodiff.
- adamnemecek 4y agoI thought you were asking about attention only transformers. This paper touches on some of it https://arxiv.org/abs/2212.10559v2 https://arxiv.org/abs/2212.10559v2.
- lostmsu 4y agoThe paper speculates that it is analogous to gradient descent and empirically confirms it is similar in behavior, but it is not a rigorous proof of any kind. The momentum experiment they made also does not seem related. E.g. it just adds past values to V, which extends the effective context length.
- adamnemecek 4y ago> but it is not a rigorous proof of any kind. Such is the nature of early theories.
- tpoacher 4y ago"Combinatorial Hopf" would make an excellent beer name! "Bartender! A half-pint of your finest Combinatorial Hopf, if you please!"
- macrolocal 4y agoFirst question: why should the attention mechanism output and residual stream match?
- adamnemecek 4y agoMatch is a bad word, the don’t match, they are duals. The residual stream aka identity mapping needs to be the identity of the attention mechanism as the attention mechanism learns. But this is the same for all residual streams, not just those in transformers. Join my discord to discuss this further https://discord.gg/mr9TAhpyBW https://discord.gg/mr9TAhpyBW
- macrolocal 4y agoWait-- the residual stream makes the attention mechanism learn the difference from the identity! Are you sure you're not thinking about auto-encoders? Edit: ok, Discord it is.
- adamnemecek 4y agoDo you see a similarity between residual stream and Dirac function?
- crosen99 4y agoI don’t believe autodiff is finding the difference in that sense. It’s finding derivatives.
- macrolocal 4y agoWell, the paper uses gradient descent to minimize that difference, like auto-encoders do.
- crosen99 4y agoGradient descent is just how neural networks (including auto-encoders) optimize parameters to minimize the loss function. They do this using derivatives to descend down the slope of the function. Autodiff is one way to compute the derivatives. Maybe we’re saying the same thing.