3 ms·
> The authors of Attention is all you need, the transformers paper, y Schmidhuber claims to have invented something formally equivalent to the linear Transform
by cma 4y ago
> The authors of Attention is all you need, the transformers paper, y
Schmidhuber claims to have invented something formally equivalent to the linear Transformer architecture (slightly weaker) years before:
https://arxiv.org/abs/2102.11174 https://arxiv.org/abs/2102.11174
- mardifoufs 4y agoSchmidhuber claims to have invented a lot of things. It's almost a running gag at this point.
- uoaei 4y agoAnd yet somehow his claims always bear some truth. I understand the comments about boys crying wolf, but it's hard to ignore the facts on the ground.
- caycep 4y agonot having a PHD in ML, it's hard for me to evaluate his claims, but how valid are all the obscure papers that he brings up? Did someone actually invent backprop in 1930 in some random corner of the former Soviet Union? Or is it a case of "true but misses the point"?
- uoaei 4y agoOften it is indeed the latter, although it is interesting that sometimes despite that it gets at the core of our contemporary understanding of the concepts in question.
- telotortium 4y ago"Formal equivalence" means very little for engineering, to be frank - the implementation is the important thing. If I wanted to be snarky, I'd say that neural networks are "formally equivalent" to Fourier analysis, which is 200 years old. I see that the paper proposes an implementation of linearized attention as well, which many others have done, but none of which seem to have caught on (although FlashAttention at least makes attention O(n) in memory, if not computation).