4 ms·

Actually, I haven’t changed my tune on this particular question since 1992 (link in the article). deep learning alone is not now and never will be the right pat

Actually, I haven’t changed my tune on this particular question since 1992 (link in the article). deep learning alone is not now and never will be the right path to strong AI. no need to change my mind about that… ps thanks for posting the article!

axiosgunnar 5y ago
Curious, what is then?
- nradov 5y ago
  No one knows. We need to do more basic research to figure that out. My guess is that it will take large scale quantum computing. But that's just speculation, I don't have any proof.
- Quarrelsome 5y ago
  brains in jars.
- pdimitar 5y ago
  Nobody truly knows, but as an outsider what's really infuriating for me is the tunnel-visioned nature of ML/DL research and investments. Especially money-wise, practically no other techniques are getting even 1% of the funding so it seems that ML/DL is some sort of a local maxima that we can't seem to crawl out of. IMO the path to a better AI very likely is tightly bound to ML/DL but to me it's obvious that they by themselves are not it. It's very likely a combination of techniques, ML/DL included.
9wzYQbTYsAIc 5y ago
> deep learning alone is not now and never will be the right path to strong AI Would you argue that “if not (deep learning, x), then not artificial consciousness” where x is any other computational technique?
jjcon 5y ago
>I haven’t changed my tune on this particular question since 1992 Well DL certainly hasn't been against a wall since '92. Pulls out phone with highly accurate voice to text, predictive keyboard, sensor activity detection, auto-categorization/labeling of photos, facial recognition, learned speech synthesis etc etc (thats just a few just in the consumer space... with no mention of government/commercial/scientific applications) They had all that in 92?
- salawat 5y ago
  Yes. In a sense. The sigmoid/neural network was well conceived, the major limiting factor was hardware and training data. ML isn't really that much further I'd wager. We've just finally gotten the computer yo catch up enough we can spit out a handful of reasonable domain specific function simulators. We're no closer to a feasible integration thereof to the point of emergent consciousness.
  - jjcon 5y ago
    >to the point of emergent consciousness But that isn't the goal nor the argument being made (not to rabbit hole in discussing that consciousness isn't even really a scientific term that can meaningfully be applied). We had some mathematical notions yes... and we have made a ton of progress since then. The perceptron doesn't hold a candle to methods today, not even close... though yes it is a building block for the field. I don't know how that could be all that controversial.
- godelski 5y ago
  He's really talking about a path towards AGI, not ML being able to do many tasks like this. There's work towards ML doing causal inference, but CI has been a major challenge for Deep Learning (a specific type of ML) and is likely not possible with it alone (see reference to hybrid models). Of course, if he were saying that ML/DL hasn't improved significantly in these recent decades then yes, he would be being dishonest. There even has been work in explicit density models, symbolic manipulation, and causal inference. All things he (presumably) cares about. But these things don't get nearly the hype nor the research power and thus is a lot slower. In the end there's really two camps. Those trying to do things and those trying to build models that understand. But note that he's using DL as a specific term and not in place of ML nor AI.
joe_the_user 5y ago
I've enjoyed your discussion of AI quite a bit over the years, especially your comments on the senselessness of GPT-3. That said, it seems to me that "possibly hitting diminishing returns" would be a better phrasing of the situation. Google's Alpha Fold is considered a serious advance in the field of protein folding. Deep learning has aided astronomers find a variety of things. etc.
deleted 5y ago
[deleted]
anu7df 5y ago
Disregarding the negative comments here, I found the article to be very much in line with my experience as a scientist in a different field working with deep learning for solving practical problems (reducing compute needed for physics, PDE constrained inverse problems) for a few years. I am not a deep learning detractor, nor am I a fan boy. Increasingly we have found that using hybrid methods: deep learning to extract parameters from large amount of data but then using the said parameters in a physics based method task executor guards against nonsense results. Small number (and dimensionality ) of the deep learning extracted parameters help with reasoning on "is deep learning still working as expected with this data" and continuing from there is usually safe. Even when the whole workflow is deep learning based, we have had to clearly reason out the domain, range and form for activation functions for critical layers in the DL network to make it work. No amount of throw in a huge resnet, transformer, FNO, chimera would do the trick without conscious thought on what the network is supposed to do. I would argue a lot of useful deep learning in hitherto un explored applications will need to have the symbolic manipulations encoded in the structure of the network.

axiosgunnar 5y ago

Curious, what is then?

nradov 5y ago
No one knows. We need to do more basic research to figure that out. My guess is that it will take large scale quantum computing. But that's just speculation, I don't have any proof.
Quarrelsome 5y ago
brains in jars.
pdimitar 5y ago
Nobody truly knows, but as an outsider what's really infuriating for me is the tunnel-visioned nature of ML/DL research and investments. Especially money-wise, practically no other techniques are getting even 1% of the funding so it seems that ML/DL is some sort of a local maxima that we can't seem to crawl out of. IMO the path to a better AI very likely is tightly bound to ML/DL but to me it's obvious that they by themselves are not it. It's very likely a combination of techniques, ML/DL included.

9wzYQbTYsAIc 5y ago

> deep learning alone is not now and never will be the right path to strong AI Would you argue that “if not (deep learning, x), then not artificial consciousness” where x is any other computational technique?

jjcon 5y ago

>I haven’t changed my tune on this particular question since 1992 Well DL certainly hasn't been against a wall since '92. Pulls out phone with highly accurate voice to text, predictive keyboard, sensor activity detection, auto-categorization/labeling of photos, facial recognition, learned speech synthesis etc etc (thats just a few just in the consumer space... with no mention of government/commercial/scientific applications) They had all that in 92?

salawat 5y ago
Yes. In a sense. The sigmoid/neural network was well conceived, the major limiting factor was hardware and training data. ML isn't really that much further I'd wager. We've just finally gotten the computer yo catch up enough we can spit out a handful of reasonable domain specific function simulators. We're no closer to a feasible integration thereof to the point of emergent consciousness.
- jjcon 5y ago
  >to the point of emergent consciousness But that isn't the goal nor the argument being made (not to rabbit hole in discussing that consciousness isn't even really a scientific term that can meaningfully be applied). We had some mathematical notions yes... and we have made a ton of progress since then. The perceptron doesn't hold a candle to methods today, not even close... though yes it is a building block for the field. I don't know how that could be all that controversial.
godelski 5y ago
He's really talking about a path towards AGI, not ML being able to do many tasks like this. There's work towards ML doing causal inference, but CI has been a major challenge for Deep Learning (a specific type of ML) and is likely not possible with it alone (see reference to hybrid models). Of course, if he were saying that ML/DL hasn't improved significantly in these recent decades then yes, he would be being dishonest. There even has been work in explicit density models, symbolic manipulation, and causal inference. All things he (presumably) cares about. But these things don't get nearly the hype nor the research power and thus is a lot slower. In the end there's really two camps. Those trying to do things and those trying to build models that understand. But note that he's using DL as a specific term and not in place of ML nor AI.

joe_the_user 5y ago

I've enjoyed your discussion of AI quite a bit over the years, especially your comments on the senselessness of GPT-3. That said, it seems to me that "possibly hitting diminishing returns" would be a better phrasing of the situation. Google's Alpha Fold is considered a serious advance in the field of protein folding. Deep learning has aided astronomers find a variety of things. etc.

deleted 5y ago

[deleted]

anu7df 5y ago

Disregarding the negative comments here, I found the article to be very much in line with my experience as a scientist in a different field working with deep learning for solving practical problems (reducing compute needed for physics, PDE constrained inverse problems) for a few years. I am not a deep learning detractor, nor am I a fan boy. Increasingly we have found that using hybrid methods: deep learning to extract parameters from large amount of data but then using the said parameters in a physics based method task executor guards against nonsense results. Small number (and dimensionality ) of the deep learning extracted parameters help with reasoning on "is deep learning still working as expected with this data" and continuing from there is usually safe. Even when the whole workflow is deep learning based, we have had to clearly reason out the domain, range and form for activation functions for critical layers in the DL network to make it work. No amount of throw in a huge resnet, transformer, FNO, chimera would do the trick without conscious thought on what the network is supposed to do. I would argue a lot of useful deep learning in hitherto un explored applications will need to have the symbolic manipulations encoded in the structure of the network.