5 ms·

background from the great Genius Makers history of ML: The press needed heroes for its AI narrative. It chose Hinton, LeCun, Bengio, and sometimes Ng, thanks l

background from the great Genius Makers history of ML: The press needed heroes for its AI narrative. It chose Hinton, LeCun, Bengio, and sometimes Ng, thanks largely to the promotional efforts of Google and Facebook. The narrative did not extend to Jürgen Schmidhuber, the German researcher based on Lake Lugano who carried the torch for neural networks in Europe during the 1990s and 2000s. Some took issue with Schmidhuber's exclusion, including Schmidhuber. In 2005, he and Alex Graves, the researcher who later joined DeepMind, had published a paper describing a speech recognition system based on LSTMs-neural networks with short-term memory. "This is that crazy Schmidhuber stuff," Hinton had told himself, "but it actually works." Now the technology was powering speech services at companies like Google and Microsoft, and Schmidhuber wanted his due. After Hinton, LeCun, and Bengio published a paper on the rise of deep learning in Nature, he wrote a critique arguing that "the Canadians" weren't as influential as they influence would soon reverberate across the industry-Schmidhuber stood up in the audience and chastised him for not citing similar work in Switzerland from the 1990s. He did this kind of thing so often, it became its own verb-as in: "You have been Schmidhubered."

nabla9 4y ago
Reading some Schmidhuber's writings on this issue is a great way to learn the history and development of AI and ANN's that is often missed. At the same time Schmidhuber's tone and anal-retentive attitude explains why he is considered so annoying. The field is full of huge egos and overly simplified narratives from a singular point of view. For example, it's true that Hinton was a torchbearer of neural networks for several decades and was off mainstream. That gives him well earned legendary status. But it's also true that other methods worked better until early or mid 2000s and had more rigorous theory behind it (theory behind older methods is still better, nn is still mostly experimenting and wandering). Hinton also had position and funding to do pioneering research and collect students all over the world. I remember reading his product of experts paper as a student in 2002 when I worked with boosting and bagging, so clearly he was not completely out of sight.
- fxtentacle 4y ago
  I believe it's reasonable for him to request a citation if significant results of his works are quoted. LeCun would probably also be upset if he read about his work in the newspaper, but mis-attributed to someone else. Also, millions in funding money depend on stuff like this. If LeCun can pretend to be the inventor, he gets the grant money. If Schmidhuber gets the citation he seeks, he can use that to apply for new funding. So it's not just prestige, these citation issues have a large monetary impact, too.
  - nabla9 4y ago
    [LEC22a] is a position paper. LeCun is expressing his view of how to go forward. It's not a technical paper describing something that works. Ideas he describe have been floating around decades and are considered good ideas, except nobody has made them work, thus the position paper for the future. Schmidhuber points out everything I said above. JEPA is old idea, etc. He also points out that the paper is "tabloid" article. Schmidhuber does not deserve being quoted for general ideas because he wrote paper of few of them.
- JohnHaugeland 4y ago
  > At the same time Schmidhuber's tone and anal-retentive attitude explains why he is considered so annoying. The people calling him annoying are generally hangers-on trying to display that they're part of the group - people who took a class once 20 years ago and think they need now to come explain. He's a lovely man, and he's 100% correct to be angry that people are getting rich and famous while not citing him. It's an ethical disaster. I really wish HN people wouldn't pile onto him this way. . > The field is full of huge egos and overly simplified narratives from a singular point of view. Stop it. Primacy in publishing is a core topic in science. If these people were at universities, and not Google or Facebook, they'd have been fired for what they're doing. Schmidthuber doesn't actually have a huge ego. He's just saying "guys, stop re-publishing my papers and claiming you came up with it." You should really be deeply disgusted in Hinton right now, not explaining Schmidthuber's ego. Hinton is being a plagiarist *and he knows it*. This isn't an accident and it isn't the first time. Hinton is committing fraud, according to academic doctrine. This is an automatic termination at most universities on Earth, even through tenure. It's time to stop making excuses for these rich men cherry picking other peoples' work for their and their employers' gain. Schmidthuber is, if anything, being remarkably calm and polite, for having been robbed blind by megacorporations of credit and wealth for decades.
dchichkov 4y ago
In my opinion the difference that Hinton students made was using GPUs to run deep learning models. First in 2010, when publishing Theano at NIPS, making deep learning on a single computer node with a large GPUs very approachable ( using a Python framework, precursor to TensorFlow). Then in 2012, with Alex Krizhevsky masterfully using two GPUs to build the landmark model AlexNet, which outperformed traditional computer vision models and was easy to use in practice.
- ekleraki 4y ago
  Funny ... According to [1], the papers [42,43] showed that plain backprop was capable of solving complex tasks without pretraining on a GPU. From [1]: > Our team then showed that good old backpropagation [A1] on GPUs (with training pattern distortions [42,43] but without any unsupervised pre-training) I can't verify since I have a barely stable connection and little battery, but somebody else should be able to. [1] https://people.idsia.ch/~juergen/firstdeeplearner.html https://people.idsia.ch/~juergen/firstdeeplearner.html [42] H. Baird. Document image defect models. IAPR Workshop, Syntactic & Structural Pattern Recognition, p 38-46, 1990 [43] P. Y. Simard, D. Steinkraus, J.C. Platt. Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis. ICDAR 2003, p 958-962, 2003.
  - catgary 4y ago
    Isn’t backprop just reverse-mode AD, which has been around since the 60’s? I get the impression that a lot of physicists and scientific computing researchers have been rolling their eyes at various ML “breakthroughs” over the years.
    - ekleraki 4y ago
      My point was not backprop, but time of publication. However, "reverse-mode" AD is not as simple as just caching results. > BP’s continuous form was derived in the early 1960s (Kelley, 1960; Bryson, 1961; Bryson and Ho, 1969). Dreyfus (1962) published the elegant derivation of BP based on the chain rule only. > explicit, efficient error backpropagation (BP) in arbitrary, discrete, possibly sparsely connected, NN-like networks But that wasn't close enough to current BP > BP’s modern efficient version for discrete sparse networks (including FORTRAN code) was published by Linnainmaa (1970). Here the complexity of computing the derivatives of the output error with respect to each weight is proportional to the number of weights. That’s the method still used today. See [1] for more details. I do not know the exact implementation details of Linnainmaa's BP, but there are all sorts of improvements to plain reverse mode auto-grad through stuff like Jacobian-Vector products that save an enormous amount of memory. Nocedal's "Numerical Optimization" covers a great deal on how to implement autograd, and I can easily recommend it to anyone interested. [1] https://www.reddit.com/r/MachineLearning/comments/e5vzun/d_jurgen_schmidhuber_on_seppo_linnainmaa_inventor/ https://www.reddit.com/r/MachineLearning/comments/e5vzun/d_j...
seydor 4y ago
Anyone who had a passing interest in ANNs in the 90s would not miss Hinton or Sejnowski or older work of Rosenblatt or Kohonen etc. They are a different league. TBH i did not encounter schmidhuber or bengio or lecun (the latter only in passing)
evolvingstuff 4y ago
Back in the early 2000's (when I first started getting into this stuff), Schmidhuber's ideas were absolutely off-the-charts revolutionary compared to anything any other lab was doing. The level of creativity coming out of IDSIA was insane. I truly believe he has not been given even slightly the level of recognition he deserves. Many of the original founders of DeepMind are his direct students. E.g. Alex Graves, who is basically the pioneer of "attention" in neural networks (and we all know what architecture that led to...)

nabla9 4y ago

Reading some Schmidhuber's writings on this issue is a great way to learn the history and development of AI and ANN's that is often missed. At the same time Schmidhuber's tone and anal-retentive attitude explains why he is considered so annoying. The field is full of huge egos and overly simplified narratives from a singular point of view. For example, it's true that Hinton was a torchbearer of neural networks for several decades and was off mainstream. That gives him well earned legendary status. But it's also true that other methods worked better until early or mid 2000s and had more rigorous theory behind it (theory behind older methods is still better, nn is still mostly experimenting and wandering). Hinton also had position and funding to do pioneering research and collect students all over the world. I remember reading his product of experts paper as a student in 2002 when I worked with boosting and bagging, so clearly he was not completely out of sight.

fxtentacle 4y ago
I believe it's reasonable for him to request a citation if significant results of his works are quoted. LeCun would probably also be upset if he read about his work in the newspaper, but mis-attributed to someone else. Also, millions in funding money depend on stuff like this. If LeCun can pretend to be the inventor, he gets the grant money. If Schmidhuber gets the citation he seeks, he can use that to apply for new funding. So it's not just prestige, these citation issues have a large monetary impact, too.
- nabla9 4y ago
  [LEC22a] is a position paper. LeCun is expressing his view of how to go forward. It's not a technical paper describing something that works. Ideas he describe have been floating around decades and are considered good ideas, except nobody has made them work, thus the position paper for the future. Schmidhuber points out everything I said above. JEPA is old idea, etc. He also points out that the paper is "tabloid" article. Schmidhuber does not deserve being quoted for general ideas because he wrote paper of few of them.
JohnHaugeland 4y ago
> At the same time Schmidhuber's tone and anal-retentive attitude explains why he is considered so annoying. The people calling him annoying are generally hangers-on trying to display that they're part of the group - people who took a class once 20 years ago and think they need now to come explain. He's a lovely man, and he's 100% correct to be angry that people are getting rich and famous while not citing him. It's an ethical disaster. I really wish HN people wouldn't pile onto him this way. . > The field is full of huge egos and overly simplified narratives from a singular point of view. Stop it. Primacy in publishing is a core topic in science. If these people were at universities, and not Google or Facebook, they'd have been fired for what they're doing. Schmidthuber doesn't actually have a huge ego. He's just saying "guys, stop re-publishing my papers and claiming you came up with it." You should really be deeply disgusted in Hinton right now, not explaining Schmidthuber's ego. Hinton is being a plagiarist *and he knows it*. This isn't an accident and it isn't the first time. Hinton is committing fraud, according to academic doctrine. This is an automatic termination at most universities on Earth, even through tenure. It's time to stop making excuses for these rich men cherry picking other peoples' work for their and their employers' gain. Schmidthuber is, if anything, being remarkably calm and polite, for having been robbed blind by megacorporations of credit and wealth for decades.

dchichkov 4y ago

In my opinion the difference that Hinton students made was using GPUs to run deep learning models. First in 2010, when publishing Theano at NIPS, making deep learning on a single computer node with a large GPUs very approachable ( using a Python framework, precursor to TensorFlow). Then in 2012, with Alex Krizhevsky masterfully using two GPUs to build the landmark model AlexNet, which outperformed traditional computer vision models and was easy to use in practice.

ekleraki 4y ago
Funny ... According to [1], the papers [42,43] showed that plain backprop was capable of solving complex tasks without pretraining on a GPU. From [1]: > Our team then showed that good old backpropagation [A1] on GPUs (with training pattern distortions [42,43] but without any unsupervised pre-training) I can't verify since I have a barely stable connection and little battery, but somebody else should be able to. [1] https://people.idsia.ch/~juergen/firstdeeplearner.html https://people.idsia.ch/~juergen/firstdeeplearner.html [42] H. Baird. Document image defect models. IAPR Workshop, Syntactic & Structural Pattern Recognition, p 38-46, 1990 [43] P. Y. Simard, D. Steinkraus, J.C. Platt. Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis. ICDAR 2003, p 958-962, 2003.
- catgary 4y ago
  Isn’t backprop just reverse-mode AD, which has been around since the 60’s? I get the impression that a lot of physicists and scientific computing researchers have been rolling their eyes at various ML “breakthroughs” over the years.
  - ekleraki 4y ago
    My point was not backprop, but time of publication. However, "reverse-mode" AD is not as simple as just caching results. > BP’s continuous form was derived in the early 1960s (Kelley, 1960; Bryson, 1961; Bryson and Ho, 1969). Dreyfus (1962) published the elegant derivation of BP based on the chain rule only. > explicit, efficient error backpropagation (BP) in arbitrary, discrete, possibly sparsely connected, NN-like networks But that wasn't close enough to current BP > BP’s modern efficient version for discrete sparse networks (including FORTRAN code) was published by Linnainmaa (1970). Here the complexity of computing the derivatives of the output error with respect to each weight is proportional to the number of weights. That’s the method still used today. See [1] for more details. I do not know the exact implementation details of Linnainmaa's BP, but there are all sorts of improvements to plain reverse mode auto-grad through stuff like Jacobian-Vector products that save an enormous amount of memory. Nocedal's "Numerical Optimization" covers a great deal on how to implement autograd, and I can easily recommend it to anyone interested. [1] https://www.reddit.com/r/MachineLearning/comments/e5vzun/d_jurgen_schmidhuber_on_seppo_linnainmaa_inventor/ https://www.reddit.com/r/MachineLearning/comments/e5vzun/d_j...

seydor 4y ago

Anyone who had a passing interest in ANNs in the 90s would not miss Hinton or Sejnowski or older work of Rosenblatt or Kohonen etc. They are a different league. TBH i did not encounter schmidhuber or bengio or lecun (the latter only in passing)

evolvingstuff 4y ago

Back in the early 2000's (when I first started getting into this stuff), Schmidhuber's ideas were absolutely off-the-charts revolutionary compared to anything any other lab was doing. The level of creativity coming out of IDSIA was insane. I truly believe he has not been given even slightly the level of recognition he deserves. Many of the original founders of DeepMind are his direct students. E.g. Alex Graves, who is basically the pioneer of "attention" in neural networks (and we all know what architecture that led to...)