4 ms·
I thought you were asking about attention only transformers. This paper touches on some of it https://arxiv.org/abs/2212.10559v2 https://arxiv.org/abs/2212.1055
by adamnemecek 4y ago
I thought you were asking about attention only transformers. This paper touches on some of it https://arxiv.org/abs/2212.10559v2 https://arxiv.org/abs/2212.10559v2.
- lostmsu 4y agoThe paper speculates that it is analogous to gradient descent and empirically confirms it is similar in behavior, but it is not a rigorous proof of any kind. The momentum experiment they made also does not seem related. E.g. it just adds past values to V, which extends the effective context length.
- adamnemecek 4y ago> but it is not a rigorous proof of any kind. Such is the nature of early theories.