5 ms·

It’s interesting that most AI models focus on learning, but in biology the heavy lifting was all done in the developmental evolution and neurogenesis. You’re no

It’s interesting that most AI models focus on learning, but in biology the heavy lifting was all done in the developmental evolution and neurogenesis. You’re not going to teach your dog relativity with back propagation—-yet we treat BP like it was going to solve all the problems in AI. Until we start focusing more on neural architecture I have no worries about AI taking over the world.

TekMol 7y ago
Well, at the moment we are not teaching dogs. We are trying to create dogs.
- rdlecler1 7y ago
  That’s not what I see.
creddit 7y ago
This is a valuable observation. I think the question, though, comes down to which of these two things should be mastered first or if they should be done in parallel. In general, the explosion of successes in DL in the last few years has been driven by AD and backprop rising to the occasion to make novel architectures explorable for the first time.
olooney 7y ago
> You’re not going to teach your dog relativity with back propagation What do you mean? RNN's can be taught to do math using BP[1]. Heck, you can do a simple version yourself in less than an hour[2]. They can't do tensor calculus yet (AFAIK) but there doesn't seem be any reason why BP in particular would be the stumbling block; the difficulty is finding the right representation. > yet we treat BP like it was going to solve all the problems in AI Except for reinforcement learning which does not use BP to solve the Bellma equations[3], AlphaZero which uses minimax[4], AutoML based on non-gradient methods such as bayesian optimization[5], etc. Research into CNNs and RNNs for image, text, and voice processing does tend to focus on BP, but not inappropriately so considering that that approach continues to make rapid progress in those domains. > Until we start focusing more on neural architecture I have no worries about AI taking over the world. I can't tell if you're arguing for or against. Are you suggesting that we should focus more on "neural architecture" so that AI can take over the world? Or are you approving of the current narrow focus for reasons of safety? [1] https://arxiv.org/pdf/1809.08590.pdf https://arxiv.org/pdf/1809.08590.pdf [2] https://machinelearningmastery.com/learn-add-numbers-seq2seq-recurrent-neural-networks/ https://machinelearningmastery.com/learn-add-numbers-seq2seq... [3] https://towardsdatascience.com/introduction-to-various-reinforcement-learning-algorithms-i-q-learning-sarsa-dqn-ddpg-72a5e0cb6287 https://towardsdatascience.com/introduction-to-various-reinf... [4] https://www.depthfirstlearning.com/2018/AlphaGoZero https://www.depthfirstlearning.com/2018/AlphaGoZero [5] https://en.wikipedia.org/wiki/Bayesian_optimization https://en.wikipedia.org/wiki/Bayesian_optimization
- YeGoblynQueenne 7y ago
  >> RNN's can be taught to do math using BP The paper you cite and every single paper on artificial neural nets learning to "do maths" or "do arithmetic" etc that has every been published. are only showing neural nets learning [1] the results of specific operations between numbers up to a certain value. There is no work that shows neural nets learning to do arbitrary arithmetic operations on arbitrary numbers. To put it very clearly: neural nets can't learn to "do maths" in any general sense. Neural nets are infamously incapable of generalising beyond their training dataset and the OP is very reasonably skeptical of comparisons between artificial and natural neural networks, given that only the latter are known to be able to generalise from few examples seen very few times. ______________ [1] As in overfitting to.
  - kahnjw 7y ago
    >Neural nets are infamously incapable of generalising beyond their training dataset In what sense? DNNs can certainly generalize to examples outside the training set. I agree that you get no guarantees on performance for samples drawn from a distribution that is different than the training/test set. Not having guaranteed expected performance isn't the same as "incapable" of generalizing to a new distribution.
    - YeGoblynQueenne 7y ago
      Sorry for the late reply. To clarify, when I say "training dataset" I mean the entire dataset used for training. Not the training partition in a cross-validation training/test split. Neural nets can interpolate between the data points in their training dataset, but cannot extrapolate to cover data points outside this dataset. This is not a matter of architecture. DNNs are no exception. Neural nets are just very bad at generalising. This lack of generalisation ability is why neural nets need to be trained with huge amounts of data, the more the better. Because they can't generalise, the only way to get them to recognise more instances of a class is to show them more examples of that class. Here's a longer discussion of this: https://blog.keras.io/the-limitations-of-deep-learning.html https://blog.keras.io/the-limitations-of-deep-learning.html

TekMol 7y ago

Well, at the moment we are not teaching dogs. We are trying to create dogs.

rdlecler1 7y ago
That’s not what I see.

creddit 7y ago

This is a valuable observation. I think the question, though, comes down to which of these two things should be mastered first or if they should be done in parallel. In general, the explosion of successes in DL in the last few years has been driven by AD and backprop rising to the occasion to make novel architectures explorable for the first time.

olooney 7y ago

> You’re not going to teach your dog relativity with back propagation What do you mean? RNN's can be taught to do math using BP[1]. Heck, you can do a simple version yourself in less than an hour[2]. They can't do tensor calculus yet (AFAIK) but there doesn't seem be any reason why BP in particular would be the stumbling block; the difficulty is finding the right representation. > yet we treat BP like it was going to solve all the problems in AI Except for reinforcement learning which does not use BP to solve the Bellma equations[3], AlphaZero which uses minimax[4], AutoML based on non-gradient methods such as bayesian optimization[5], etc. Research into CNNs and RNNs for image, text, and voice processing does tend to focus on BP, but not inappropriately so considering that that approach continues to make rapid progress in those domains. > Until we start focusing more on neural architecture I have no worries about AI taking over the world. I can't tell if you're arguing for or against. Are you suggesting that we should focus more on "neural architecture" so that AI can take over the world? Or are you approving of the current narrow focus for reasons of safety? [1] https://arxiv.org/pdf/1809.08590.pdf https://arxiv.org/pdf/1809.08590.pdf [2] https://machinelearningmastery.com/learn-add-numbers-seq2seq-recurrent-neural-networks/ https://machinelearningmastery.com/learn-add-numbers-seq2seq... [3] https://towardsdatascience.com/introduction-to-various-reinforcement-learning-algorithms-i-q-learning-sarsa-dqn-ddpg-72a5e0cb6287 https://towardsdatascience.com/introduction-to-various-reinf... [4] https://www.depthfirstlearning.com/2018/AlphaGoZero https://www.depthfirstlearning.com/2018/AlphaGoZero [5] https://en.wikipedia.org/wiki/Bayesian_optimization https://en.wikipedia.org/wiki/Bayesian_optimization

YeGoblynQueenne 7y ago
>> RNN's can be taught to do math using BP The paper you cite and every single paper on artificial neural nets learning to "do maths" or "do arithmetic" etc that has every been published. are only showing neural nets learning [1] the results of specific operations between numbers up to a certain value. There is no work that shows neural nets learning to do arbitrary arithmetic operations on arbitrary numbers. To put it very clearly: neural nets can't learn to "do maths" in any general sense. Neural nets are infamously incapable of generalising beyond their training dataset and the OP is very reasonably skeptical of comparisons between artificial and natural neural networks, given that only the latter are known to be able to generalise from few examples seen very few times. ______________ [1] As in overfitting to.
- kahnjw 7y ago
  >Neural nets are infamously incapable of generalising beyond their training dataset In what sense? DNNs can certainly generalize to examples outside the training set. I agree that you get no guarantees on performance for samples drawn from a distribution that is different than the training/test set. Not having guaranteed expected performance isn't the same as "incapable" of generalizing to a new distribution.
  - YeGoblynQueenne 7y ago
    Sorry for the late reply. To clarify, when I say "training dataset" I mean the entire dataset used for training. Not the training partition in a cross-validation training/test split. Neural nets can interpolate between the data points in their training dataset, but cannot extrapolate to cover data points outside this dataset. This is not a matter of architecture. DNNs are no exception. Neural nets are just very bad at generalising. This lack of generalisation ability is why neural nets need to be trained with huge amounts of data, the more the better. Because they can't generalise, the only way to get them to recognise more instances of a class is to show them more examples of that class. Here's a longer discussion of this: https://blog.keras.io/the-limitations-of-deep-learning.html https://blog.keras.io/the-limitations-of-deep-learning.html