8 ms·

Yann LeCun vs. Christopher Manning on Need for Priors in Deep Learning

150 points by andreyk 9y ago

nextos 9y ago
Can anyone point to any reference where models depart from "closely-supervised end-to-end learning"? In the article, LeCun & Manning argue this paradigm has some limitations, and I do agree. I'm thinking the field will evolve to systems becoming a combination of probabilistic logic-based engines (which represent formal causal reasoning) plus lots of deep models (which represent intuition, hypothesis generation and specialized tasks like vision).
- make3 9y ago
  "Unsupervised Neural Machine Translation", https://arxiv.org/abs/1710.11041 https://arxiv.org/abs/1710.11041
- nl 9y ago
  There are lots of examples. In the NLP space, Word2Vec and other embeddding models learn analogy relationships in an unsupervised way. Things like the OpenAI unsupervised sentiment detection model[1] show there is a lot more that can be done in this area. [1] https://blog.openai.com/unsupervised-sentiment-neuron/ https://blog.openai.com/unsupervised-sentiment-neuron/
- f00_ 9y ago
  Generative Adversarial Networks (GAN) and Autoencoders are the ones that come to mind. There are also all the models used in research leading up to the "deep learning" craze: Helmholtz machine, Boltzman machine, Deep belief networks. http://pyro.ai/examples/vae.html http://pyro.ai/examples/vae.html http://www.scholarpedia.org/article/Boltzmann_machine http://www.scholarpedia.org/article/Boltzmann_machine http://www.scholarpedia.org/article/Deep_belief_networks http://www.scholarpedia.org/article/Deep_belief_networks Geoff Hinton's Coursera course goes pretty deep on the unsupervised models https://www.coursera.org/learn/neural-networks https://www.coursera.org/learn/neural-networks Reinforcement learning probably counts too (Deep Q Learning, Policy Gradient, Actor-Critic networks might be equivalent to GANs?) http://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html http://pytorch.org/tutorials/intermediate/reinforcement_q_le... https://github.com/vy007vikas/PyTorch-ActorCriticRL https://github.com/vy007vikas/PyTorch-ActorCriticRL
kevinwang 9y ago
>As an example (28:57), he described how the human brain does not have any innate convolutional structure – but it doesn’t need to, because as an effective unsupervised learner, the brain can learn the same low-level image features (e.g. oriented edge detectors) as a ConvNet, even without the convolutional weight-sharing constraint. I think the first part of the sentence should be that the brain doesn't have an innate weight sharing (like is stated in the end of the sentence), not that it is not convolutional. I believe the convolutional structure is actually copied from the visual cortex (but with no weight sharing as far as we know)
- yorwba 9y ago
  Convolutions are mathematically defined by application of the same kernel at each possible position. If that kernel has finite support, you also get locality. The visual cortex has locality, but without weight-sharing between functionally identical neurons, it's not convolutional.
  - empiricus 9y ago
    The visual cortex neurons have not only locality, but also they learn/recognize similar features. This was the original inspiration for the conv nets: it was a way to learn efficient local features and apply them to the whole image. To me it seems that cortex neurons and the initial layers of conv nets do similar work, but with constraints of their implementations: biological neurons cannot share weights, and for artificial neurons it is more efficient to learn and compute dense convolution.
    - posterboy 9y ago
      Disclaimer: I'm learning deep learning mainly from HN comments and just want to provoke more insights. I have no idea what weight sharing is or how kernels are represented in networks, but I do know that e.g. a blur filter or edge filter is represented as convolution matrix. It is dangerously confusing to reapply the neural-net-terminology to neuronal-nets isn't it? The weight of a kernel of biological neurons, what is that supposed to mean? If you haven't stopped reading yet, please consider: in case, as I have to assume, you mean there is a specific ensemble of neurons that represents a kernel of given weights corresponding to exactly one area of retina, then isn't sharing between "pixels" achieved simply by the eye's jittering? For better or worse, assume I'm the adversary in a GAN and ignore me if it doesn't make sense.
YeGoblynQueenne 9y ago
>> Over the last few decades, innate priors have gone out of fashion, and today Deep Learning research prizes closely-supervised end-to-end learning (supported by big-data and big-compute) as the dominant paradigm. I don't get why deep learning researchers are so hung up on learning everything from scratch. The tendency for ever more compute and data is just unsustainable. For problems like natural language, where distinct evens may be infinite, you can keep throwing data at the problem and you'll never make a dent to it. There are problems that grow at a pace that cannot be matched by any computer, no matter how powerful. What's more, as a civilisation we have nothing if not knowledge about the world. We have been accumulating it for thousands of years. It's what makes the difference between an intelligent human and an educated intelligent human. And it's a big difference. So, if we have all this background knowledge, why not use it, and make our lives easier?
- SiempreViernes 9y ago
  I guess they never got a visit from Minsky, and thus haven’t reached enligthenment. ======================= Sussman attains enlightenment In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6. “What are you doing?”, asked Minsky. “I am training a randomly wired neural net to play Tic-Tac-Toe” Sussman replied. “Why is the net wired randomly?”, asked Minsky. “I do not want it to have any preconceptions of how to play”, Sussman said. Minsky then shut his eyes. “Why do you close your eyes?”, Sussman asked his teacher. “So that the room will be empty.” At that moment, Sussman was enlightened ==================== http://spetharrific.tumblr.com/post/26600309788/sussman-attains-enlightenment http://spetharrific.tumblr.com/post/26600309788/sussman-atta...
  - smallnamespace 9y ago
    I think it's a pretty well established in learning theory that the combination of neural network topology + the training procedure sets up an implicit prior (it establishes a bias towards the type of things the network will learn more easily from the input data). In fact, this is a corollary of the No Free Lunch Theorem. The problem is that these are extremely high-dimensional spaces, and even narrowing down the space within which the prior might be defined (much less finding the prior itself) is mathematically difficult. Hence the progress is slow and halting—it's consisted of discussions like what the representations fully-trained neural networks find within the early activation layers, which means the question is being investigated empirically, without the benefit of a fully-fledged theory. Initeresting, the 'nature vs. nurture' debate can be viewed as basically a discussion of learning priors in the context of human beings. Turns out the truth is pretty subtle—for many problem domains, humans (and animals) can be made to learn a very large set of things with repeated training, but we also have strong priors, so some things can be taught much more easily and quickly than others. Our brain networks are primed to learn certain things, e.g. ducklings have a prior that the first animate object they see is their mother. It's possible to force a duckling to unlearn its maternal imprinting, but only with repeated and large amounts of negative conditioning. It seems pretty likely that this prior is embedded in both the topology and neurochemical functioning of a normally developing brain.
micro_cam 9y ago
Just started watching the video so they may mention this but one thing I find fascinating is that some recent work suggests that the optimization algorithm (usually stochastic gradient descent) or complexity of the loss surface (ie having lots of local minima that are almost as good as the global maxima) may actually be seen to induce a kind of regularization prior. Ie these seemingly really complex models are actually biased to find simpler solutions that generalize well in a way that turns out to often work better then trying to explicitly learn a simple model.

150 points by andreyk 9y ago

Can anyone point to any reference where models depart from "closely-supervised end-to-end learning"? In the article, LeCun & Manning argue this paradigm has some limitations, and I do agree. I'm thinking the field will evolve to systems becoming a combination of probabilistic logic-based engines (which represent formal causal reasoning) plus lots of deep models (which represent intuition, hypothesis generation and specialized tasks like vision).

make3 9y ago
"Unsupervised Neural Machine Translation", https://arxiv.org/abs/1710.11041 https://arxiv.org/abs/1710.11041
nl 9y ago
There are lots of examples. In the NLP space, Word2Vec and other embeddding models learn analogy relationships in an unsupervised way. Things like the OpenAI unsupervised sentiment detection model[1] show there is a lot more that can be done in this area. [1] https://blog.openai.com/unsupervised-sentiment-neuron/ https://blog.openai.com/unsupervised-sentiment-neuron/
f00_ 9y ago
Generative Adversarial Networks (GAN) and Autoencoders are the ones that come to mind. There are also all the models used in research leading up to the "deep learning" craze: Helmholtz machine, Boltzman machine, Deep belief networks. http://pyro.ai/examples/vae.html http://pyro.ai/examples/vae.html http://www.scholarpedia.org/article/Boltzmann_machine http://www.scholarpedia.org/article/Boltzmann_machine http://www.scholarpedia.org/article/Deep_belief_networks http://www.scholarpedia.org/article/Deep_belief_networks Geoff Hinton's Coursera course goes pretty deep on the unsupervised models https://www.coursera.org/learn/neural-networks https://www.coursera.org/learn/neural-networks Reinforcement learning probably counts too (Deep Q Learning, Policy Gradient, Actor-Critic networks might be equivalent to GANs?) http://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html http://pytorch.org/tutorials/intermediate/reinforcement_q_le... https://github.com/vy007vikas/PyTorch-ActorCriticRL https://github.com/vy007vikas/PyTorch-ActorCriticRL

kevinwang 9y ago

>As an example (28:57), he described how the human brain does not have any innate convolutional structure – but it doesn’t need to, because as an effective unsupervised learner, the brain can learn the same low-level image features (e.g. oriented edge detectors) as a ConvNet, even without the convolutional weight-sharing constraint. I think the first part of the sentence should be that the brain doesn't have an innate weight sharing (like is stated in the end of the sentence), not that it is not convolutional. I believe the convolutional structure is actually copied from the visual cortex (but with no weight sharing as far as we know)

yorwba 9y ago
Convolutions are mathematically defined by application of the same kernel at each possible position. If that kernel has finite support, you also get locality. The visual cortex has locality, but without weight-sharing between functionally identical neurons, it's not convolutional.
- empiricus 9y ago
  The visual cortex neurons have not only locality, but also they learn/recognize similar features. This was the original inspiration for the conv nets: it was a way to learn efficient local features and apply them to the whole image. To me it seems that cortex neurons and the initial layers of conv nets do similar work, but with constraints of their implementations: biological neurons cannot share weights, and for artificial neurons it is more efficient to learn and compute dense convolution.
  - posterboy 9y ago
    Disclaimer: I'm learning deep learning mainly from HN comments and just want to provoke more insights. I have no idea what weight sharing is or how kernels are represented in networks, but I do know that e.g. a blur filter or edge filter is represented as convolution matrix. It is dangerously confusing to reapply the neural-net-terminology to neuronal-nets isn't it? The weight of a kernel of biological neurons, what is that supposed to mean? If you haven't stopped reading yet, please consider: in case, as I have to assume, you mean there is a specific ensemble of neurons that represents a kernel of given weights corresponding to exactly one area of retina, then isn't sharing between "pixels" achieved simply by the eye's jittering? For better or worse, assume I'm the adversary in a GAN and ignore me if it doesn't make sense.

YeGoblynQueenne 9y ago

>> Over the last few decades, innate priors have gone out of fashion, and today Deep Learning research prizes closely-supervised end-to-end learning (supported by big-data and big-compute) as the dominant paradigm. I don't get why deep learning researchers are so hung up on learning everything from scratch. The tendency for ever more compute and data is just unsustainable. For problems like natural language, where distinct evens may be infinite, you can keep throwing data at the problem and you'll never make a dent to it. There are problems that grow at a pace that cannot be matched by any computer, no matter how powerful. What's more, as a civilisation we have nothing if not knowledge about the world. We have been accumulating it for thousands of years. It's what makes the difference between an intelligent human and an educated intelligent human. And it's a big difference. So, if we have all this background knowledge, why not use it, and make our lives easier?

SiempreViernes 9y ago
I guess they never got a visit from Minsky, and thus haven’t reached enligthenment. ======================= Sussman attains enlightenment In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6. “What are you doing?”, asked Minsky. “I am training a randomly wired neural net to play Tic-Tac-Toe” Sussman replied. “Why is the net wired randomly?”, asked Minsky. “I do not want it to have any preconceptions of how to play”, Sussman said. Minsky then shut his eyes. “Why do you close your eyes?”, Sussman asked his teacher. “So that the room will be empty.” At that moment, Sussman was enlightened ==================== http://spetharrific.tumblr.com/post/26600309788/sussman-attains-enlightenment http://spetharrific.tumblr.com/post/26600309788/sussman-atta...
- smallnamespace 9y ago
  I think it's a pretty well established in learning theory that the combination of neural network topology + the training procedure sets up an implicit prior (it establishes a bias towards the type of things the network will learn more easily from the input data). In fact, this is a corollary of the No Free Lunch Theorem. The problem is that these are extremely high-dimensional spaces, and even narrowing down the space within which the prior might be defined (much less finding the prior itself) is mathematically difficult. Hence the progress is slow and halting—it's consisted of discussions like what the representations fully-trained neural networks find within the early activation layers, which means the question is being investigated empirically, without the benefit of a fully-fledged theory. Initeresting, the 'nature vs. nurture' debate can be viewed as basically a discussion of learning priors in the context of human beings. Turns out the truth is pretty subtle—for many problem domains, humans (and animals) can be made to learn a very large set of things with repeated training, but we also have strong priors, so some things can be taught much more easily and quickly than others. Our brain networks are primed to learn certain things, e.g. ducklings have a prior that the first animate object they see is their mother. It's possible to force a duckling to unlearn its maternal imprinting, but only with repeated and large amounts of negative conditioning. It seems pretty likely that this prior is embedded in both the topology and neurochemical functioning of a normally developing brain.

micro_cam 9y ago

Just started watching the video so they may mention this but one thing I find fascinating is that some recent work suggests that the optimization algorithm (usually stochastic gradient descent) or complexity of the loss surface (ie having lots of local minima that are almost as good as the global maxima) may actually be seen to induce a kind of regularization prior. Ie these seemingly really complex models are actually biased to find simpler solutions that generalize well in a way that turns out to often work better then trying to explicitly learn a simple model.