4 ms·

This is a little harsh. Hinton trudged along with neural networks through the coldest AI winter and helped create the conditions for OpenAI to have all the raw

This is a little harsh. Hinton trudged along with neural networks through the coldest AI winter and helped create the conditions for OpenAI to have all the raw ingredients needed to cook up something powerful.

KRAKRISMOTT 3y ago
If you need to build an airplane, would you rather consult Newton, the Wright brothers, or a modern aerospace engineer? Inventing a field and snatching up the low hanging fruits doesn't mean somebody would be able to consistently create leading edge output. Most of the advances in deep learning are due to hardware scaling, and the success of a few very specific architectures. Yes credit's due where credit's due, but academia name recognition is very much winner take all. For all the criticism Schumidhuber has received, he has a point. The authors of Attention is all you need, the transformers paper, yolo, have nowhere close to the name recognition of the Turing award trio despite generating similar if not more value through their ideas.
- cma 3y ago
  > The authors of Attention is all you need, the transformers paper, y Schmidhuber claims to have invented something formally equivalent to the linear Transformer architecture (slightly weaker) years before: https://arxiv.org/abs/2102.11174 https://arxiv.org/abs/2102.11174
  - mardifoufs 3y ago
    Schmidhuber claims to have invented a lot of things. It's almost a running gag at this point.
    - uoaei 3y ago
      And yet somehow his claims always bear some truth. I understand the comments about boys crying wolf, but it's hard to ignore the facts on the ground.
      - caycep 3y ago
        not having a PHD in ML, it's hard for me to evaluate his claims, but how valid are all the obscure papers that he brings up? Did someone actually invent backprop in 1930 in some random corner of the former Soviet Union? Or is it a case of "true but misses the point"?
        uoaei 3y ago
        Often it is indeed the latter, although it is interesting that sometimes despite that it gets at the core of our contemporary understanding of the concepts in question.
  - telotortium 3y ago
    "Formal equivalence" means very little for engineering, to be frank - the implementation is the important thing. If I wanted to be snarky, I'd say that neural networks are "formally equivalent" to Fourier analysis, which is 200 years old. I see that the paper proposes an implementation of linearized attention as well, which many others have done, but none of which seem to have caught on (although FlashAttention at least makes attention O(n) in memory, if not computation).
- innagadadavida 3y ago
  There are multiple dimensions here - fame and fortune at the very least and whether it is localized or global in scope. It is still winner takes all, but if you look at the overall landscape, there are plenty of opportunities where you can have an outsized impact - you can have localized fame and fortune (anyone with AI expertise under their belt have no problems with fortune!)

KRAKRISMOTT 3y ago

If you need to build an airplane, would you rather consult Newton, the Wright brothers, or a modern aerospace engineer? Inventing a field and snatching up the low hanging fruits doesn't mean somebody would be able to consistently create leading edge output. Most of the advances in deep learning are due to hardware scaling, and the success of a few very specific architectures. Yes credit's due where credit's due, but academia name recognition is very much winner take all. For all the criticism Schumidhuber has received, he has a point. The authors of Attention is all you need, the transformers paper, yolo, have nowhere close to the name recognition of the Turing award trio despite generating similar if not more value through their ideas.

cma 3y ago
> The authors of Attention is all you need, the transformers paper, y Schmidhuber claims to have invented something formally equivalent to the linear Transformer architecture (slightly weaker) years before: https://arxiv.org/abs/2102.11174 https://arxiv.org/abs/2102.11174
- mardifoufs 3y ago
  Schmidhuber claims to have invented a lot of things. It's almost a running gag at this point.
  - uoaei 3y ago
    And yet somehow his claims always bear some truth. I understand the comments about boys crying wolf, but it's hard to ignore the facts on the ground.
    - caycep 3y ago
      not having a PHD in ML, it's hard for me to evaluate his claims, but how valid are all the obscure papers that he brings up? Did someone actually invent backprop in 1930 in some random corner of the former Soviet Union? Or is it a case of "true but misses the point"?
      - uoaei 3y ago
        Often it is indeed the latter, although it is interesting that sometimes despite that it gets at the core of our contemporary understanding of the concepts in question.
- telotortium 3y ago
  "Formal equivalence" means very little for engineering, to be frank - the implementation is the important thing. If I wanted to be snarky, I'd say that neural networks are "formally equivalent" to Fourier analysis, which is 200 years old. I see that the paper proposes an implementation of linearized attention as well, which many others have done, but none of which seem to have caught on (although FlashAttention at least makes attention O(n) in memory, if not computation).
innagadadavida 3y ago
There are multiple dimensions here - fame and fortune at the very least and whether it is localized or global in scope. It is still winner takes all, but if you look at the overall landscape, there are plenty of opportunities where you can have an outsized impact - you can have localized fame and fortune (anyone with AI expertise under their belt have no problems with fortune!)