4 ms·

Really? Hinton dont need openAI to be relevant. He literally invented back propagation. He sticked to deep learning through 1990s and 2000s when almost all majo

Really? Hinton dont need openAI to be relevant. He literally invented back propagation. He sticked to deep learning through 1990s and 2000s when almost all major scientist abandoned it. He was using neural networks for language model in 2007-08 when no one knew what it was. Again the deep learning in 2010s started when his students created AlexNet by coding deep learning in GPU. Chief Scientist of OpenAI Ilya Sutskever was one of his student while developing the paper. He already have a Turing award and don't give a rat's ass about who owns how much search traffic. OpenAI just like Google will give him millions of dollar just to be a part of organization

PartiallyTyped 3y ago
Hinton didn’t invent back prop. > Explicit, efficient error backpropagation (BP) in arbitrary, discrete, possibly sparsely connected, NN-like networks apparently was first described in a 1970 master's thesis (Linnainmaa, 1970, 1976), albeit without reference to NNs. BP is also known as the reverse mode of automatic differentiation (e.g., Griewank, 2012), where the costs of forward activation spreading essentially equal the costs of backward derivative calculation. See early BP FORTRAN code (Linnainmaa, 1970) and closely related work (Ostrovskii et al., 1971). > BP was soon explicitly used to minimize cost functions by adapting control parameters (weights) (Dreyfus, 1973). This was followed by some preliminary, NN-specific discussion (Werbos, 1974, section 5.5.1), and a computer program for automatically deriving and implementing BP for any given differentiable system (Speelpenning, 1980). > To my knowledge, the first NN-specific application of efficient BP as above was described by Werbos (1982). Related work was published several years later (Parker, 1985; LeCun, 1985). When computers had become 10,000 times faster per Dollar and much more accessible than those of 1960-1970, a paper of 1986 significantly contributed to the popularisation of BP for NNs (Rumelhart et al., 1986), experimentally demonstrating the emergence of useful internal representations in hidden layers. https://people.idsia.ch/~juergen/who-invented-backpropagation-2014.html https://people.idsia.ch/~juergen/who-invented-backpropagatio... Hinton wasn’t the first to use NNs for language models either. That was Bengio.
- neel8986 3y ago
  I mean he was one of the first to use backprop for training multilayer perceptron. Their experiments showed that such networks can learn useful internal representations of data[1]. 1987. Nevertheless he is one of the founding fathers of deep learning [1]Learning representations by back-propagating errors
  - archgoon 3y ago
    [dead]
  - Kranar 3y ago
    It's really sad how poor attribution is in ML. Hinton certainly made important contributions to backpropagation, but he neither invented backpropagation nor was he even close to the first person to use it for multilayer perceptrons. You've now gone from one false claim "he literally invented backpropagation", to another false claim "he is one of the first people to use it for multilayer perceptrons", and will need to revise your claim even further. I don't particularly blame you specifically, as I said the field of ML is so bad when it comes to properly recognizing the teams of people who made significant contributions to it.
    - zo1 3y ago
      This is a marketing problem fundamentally, I'd argue. That the article or any serious piece would use a term such as "Godfather of AI" is incredibly worrying and makes me think it's pushing an agenda or is some sort of paid advertisement with extra steps to disguise it.
      - PartiallyTyped 3y ago
        I have grown an aversion, and possibly a knee-jerk reaction to such pieces. I have a lot of trouble taking them seriously, and I am inclined to give them a lot more scrutiny than otherwise.
janalsncm 3y ago
I’m not convinced that inventing back propagation gives one the authority to opine on more general technological/social trends. Frankly, many of the most important questions are difficult or impossible to know. In the case of neural networks, Hinton himself would never have become as famous were it not for one of those trends (the cost of GPU compute and the breakthrough of using GPUs for training) which was difficult or impossible to foresee. In an alternate universe, NNs are still slow and compute limited, and we use something like evolutionary algorithms for solving hard problems. Hinton would still be just as smart and backpropagation still just as sound but no one would listen to his opinions on the future of AI. The point is, he is quite lucky in terms of time and place, and giving outsized weight to his opinions on matters not directly related to his work is a fairly clear example of survivorship bias. Finally, we also shouldn’t ignore the fact that Hinton’s isn’t the only well-credentialed opinion out there. There are other equally if not more esteemed academics with whom Hinton is at odds. Him inventing backpropagation is good enough to get him in the door to that conversation, but doesn’t give him carte blanche authority on the matter.
- adamisom 3y ago
  Of course he was lucky, you should expect that in general for well-known people because selection pressures that led you to hear of them, vs not hear of them, are likely to involve luck. That is not at all a slam dunk argument. It’s barely anything.
  - janalsncm 3y ago
    Well unless you’re claiming the same luck that led to Hinton’s fame will lead to his accuracy on the much broader and less constrained topic of the relationship between automated systems and society, I don’t see how it’s not something. My main point wasn’t to undermine Hinton by saying he was lucky. I did do that and I stand by it. But my main point was to say that to a large degree the future on this issue is unknowable because it depends on so many crucial yet undetermined factors. And there’s nothing you could know about backpropagation, neural networks, or computer science in general which could resolve those questions.
usgroup 3y ago
This sort of reminds me of Bloomberg articles wherein every time there is some "black swan" event, they go and find an analyst or economist that "got it right" and he gets to be prophet for a day: never mind that said analyst/economist may have predicted 100 of the last 3 financial crashes, they were "right" about this one.

PartiallyTyped 3y ago

Hinton didn’t invent back prop. > Explicit, efficient error backpropagation (BP) in arbitrary, discrete, possibly sparsely connected, NN-like networks apparently was first described in a 1970 master's thesis (Linnainmaa, 1970, 1976), albeit without reference to NNs. BP is also known as the reverse mode of automatic differentiation (e.g., Griewank, 2012), where the costs of forward activation spreading essentially equal the costs of backward derivative calculation. See early BP FORTRAN code (Linnainmaa, 1970) and closely related work (Ostrovskii et al., 1971). > BP was soon explicitly used to minimize cost functions by adapting control parameters (weights) (Dreyfus, 1973). This was followed by some preliminary, NN-specific discussion (Werbos, 1974, section 5.5.1), and a computer program for automatically deriving and implementing BP for any given differentiable system (Speelpenning, 1980). > To my knowledge, the first NN-specific application of efficient BP as above was described by Werbos (1982). Related work was published several years later (Parker, 1985; LeCun, 1985). When computers had become 10,000 times faster per Dollar and much more accessible than those of 1960-1970, a paper of 1986 significantly contributed to the popularisation of BP for NNs (Rumelhart et al., 1986), experimentally demonstrating the emergence of useful internal representations in hidden layers. https://people.idsia.ch/~juergen/who-invented-backpropagation-2014.html https://people.idsia.ch/~juergen/who-invented-backpropagatio... Hinton wasn’t the first to use NNs for language models either. That was Bengio.

neel8986 3y ago
I mean he was one of the first to use backprop for training multilayer perceptron. Their experiments showed that such networks can learn useful internal representations of data[1]. 1987. Nevertheless he is one of the founding fathers of deep learning [1]Learning representations by back-propagating errors
- archgoon 3y ago
  [dead]
- Kranar 3y ago
  It's really sad how poor attribution is in ML. Hinton certainly made important contributions to backpropagation, but he neither invented backpropagation nor was he even close to the first person to use it for multilayer perceptrons. You've now gone from one false claim "he literally invented backpropagation", to another false claim "he is one of the first people to use it for multilayer perceptrons", and will need to revise your claim even further. I don't particularly blame you specifically, as I said the field of ML is so bad when it comes to properly recognizing the teams of people who made significant contributions to it.
  - zo1 3y ago
    This is a marketing problem fundamentally, I'd argue. That the article or any serious piece would use a term such as "Godfather of AI" is incredibly worrying and makes me think it's pushing an agenda or is some sort of paid advertisement with extra steps to disguise it.
    - PartiallyTyped 3y ago
      I have grown an aversion, and possibly a knee-jerk reaction to such pieces. I have a lot of trouble taking them seriously, and I am inclined to give them a lot more scrutiny than otherwise.

janalsncm 3y ago

I’m not convinced that inventing back propagation gives one the authority to opine on more general technological/social trends. Frankly, many of the most important questions are difficult or impossible to know. In the case of neural networks, Hinton himself would never have become as famous were it not for one of those trends (the cost of GPU compute and the breakthrough of using GPUs for training) which was difficult or impossible to foresee. In an alternate universe, NNs are still slow and compute limited, and we use something like evolutionary algorithms for solving hard problems. Hinton would still be just as smart and backpropagation still just as sound but no one would listen to his opinions on the future of AI. The point is, he is quite lucky in terms of time and place, and giving outsized weight to his opinions on matters not directly related to his work is a fairly clear example of survivorship bias. Finally, we also shouldn’t ignore the fact that Hinton’s isn’t the only well-credentialed opinion out there. There are other equally if not more esteemed academics with whom Hinton is at odds. Him inventing backpropagation is good enough to get him in the door to that conversation, but doesn’t give him carte blanche authority on the matter.

adamisom 3y ago
Of course he was lucky, you should expect that in general for well-known people because selection pressures that led you to hear of them, vs not hear of them, are likely to involve luck. That is not at all a slam dunk argument. It’s barely anything.
- janalsncm 3y ago
  Well unless you’re claiming the same luck that led to Hinton’s fame will lead to his accuracy on the much broader and less constrained topic of the relationship between automated systems and society, I don’t see how it’s not something. My main point wasn’t to undermine Hinton by saying he was lucky. I did do that and I stand by it. But my main point was to say that to a large degree the future on this issue is unknowable because it depends on so many crucial yet undetermined factors. And there’s nothing you could know about backpropagation, neural networks, or computer science in general which could resolve those questions.

usgroup 3y ago

This sort of reminds me of Bloomberg articles wherein every time there is some "black swan" event, they go and find an analyst or economist that "got it right" and he gets to be prophet for a day: never mind that said analyst/economist may have predicted 100 of the last 3 financial crashes, they were "right" about this one.