5 ms·

I did my postdoctoral research on deep learning, and got into it back when Geoff Hinton's work was just an unpublished tech report. So if anyone has any questi

by bravura 14y ago

I did my postdoctoral research on deep learning, and got into it back when Geoff Hinton's work was just an unpublished tech report. So if anyone has any questions on it, I will try to answer.

e98cuenc 14y ago
Is there any open source ML framework that includes support for DNN? Do you know of any tutorial that may guide the beginner using DNN, I have no idea how to choose the number of hidden layers and activation functions. Thanks!
- dumitrue 14y ago
  http://deeplearning.net/software/theano/ http://deeplearning.net/software/theano/ is a good place to start. It's open source, in Python, has a few tutorials that lead you towards some rather state-of-the-art methods. There's no secret sauce for how to choose the number of layers and the activation functions (and anyone that tells you otherwise is lying). It's all application-dependent and the best way to do it is by cross-validation. Can answer questions about this topic in conjunction with mr. bravura (I'm working with the code referenced in that paper by Dean et al., here at Google, and did grad school at the same place as bravura did his postdoc).
  - e98cuenc 14y ago
    I'm a xoogler, stuff like this makes me wish I was back there :) I want to play a bit with this stuff and have some fun, so thanks for the Theano reference, seems cool! WRT the DNN parameters, is it possible to try all potential possibilities (within reason) and find the best one using only cross-validation, or are there just too many choices and you have to use intuition? (from your comment I don't get if cross-validation is ok to get optimal number of layers etc. or if you have to be "smart") Thanks for replying!
    - dumitrue 14y ago
      For things like the number of layers, the total number of options is relatively small -- usually people try between 1 and 6-7. For most of the other parameters you have to be smarter than that, especially since a lot of them are real-valued, so you can't really explore them all. One of the trends these days is to perform automatic hyper-parameter tuning, especially for cases where a full exploration of hyper-parameters via grid-search would mean a combinatorial explosion of possibilities (and for neural networks you can conceivably explore dozen of hyper-parameters). A friend of mine just got a paper published at NIPS (same conference) on using Bayesian optimization/Gaussian processes for optimizing the hyper-parameters of a model -- http://www.dmi.usherb.ca/~larocheh/publications/gpopt_nips.pdf http://www.dmi.usherb.ca/~larocheh/publications/gpopt_nips.p.... They get better than state of the art results on a couple of benchmarks, which is neat. The code is public -- http://www.cs.toronto.edu/~jasper/software.html http://www.cs.toronto.edu/~jasper/software.html -- and in python, so you could potentially try it out (runs on EC2, too). Btw, Geoff Hinton is teaching an introductory neural nets class on Coursera these days, you should check it out, he's a great teacher. Also, you can always come back to Google, we're doing cool stuff with this :)
- seiji 14y ago
  Ask me again in two weeks when my online course in advanced machine learning for extreme beginners kicks off. We won't start with deep learning immediately, but it's in the pipeline not too soon after launch.
  - e98cuenc 14y ago
    I hope I'm not getting too off-topic, but will this course be taught in coursera or somewhere else? (I want to put a reminder to check this out). Thanks!
    - seiji 14y ago
      It'll be on a new site for teaching complex things in non-complicated ways. The goal is to allow everyone from clever middle school students through retired people to understand the coming changes to the world. There's a huge on-site community focus too. We don't want there to be 100,000 anonymous people just going through the motions. There will be plenty of interaction between course material and community feedback. It's kinda awesome. Topics will be presented in multiple ways (simple and intermediate) so you can have plenty of different views on the same material. The material works as both zero-knowledge intro to the topics as well as quick refreshers if you haven't seen the material in a while (quick -- what's an eigenvector?!). The launch courses will be 1.) real-world applications of probability and statistics (signal extractions), 2.) linear algebra for computer science, and 3.) wildcard (a random assortment of whatever the heck we think is important or entertaining to know). Future courses are: introduction to neural networks, introduction to computational neuroscience, introduction to deep learning, advanced deep learning, how to take over the world with a few dozen GPUs, avenues by which google will become irrelevant, and robotics for fun and evil. This is phase zero of a four phase plan. I'll get some pre-launch material together to shove down HN shortly, then it'll launch a few days later. Hopefully you'll hear about the project again.
      - stephenlee 14y ago
        Great work. It's so cool, can't wait to take part in.
ArbitraryLimits 14y ago
Please tell me whether the term "deep learning" has any meaning at all, and if so, what is it? It sounds like a buzzword to me, like "data scientist."
- dumitrue 14y ago
  To some extent it's true that "deep learning" is a buzzword, though it is less arbitrary that "data scientist" (which makes no sense whatsoever). In a nutshell, what people mean by "deep learning" is the collection of tips and tricks that make it possible to efficiently train multiple layers of non-linear hidden units (or feature extractors). Sometimes these are multi-layer neural nets, but they don't have to be.
  - fudged71 14y ago
    I'm in the middle of a machine learning class and it's all very interesting, but I've got one question for you: How difficult is it to make a neural net "forget"? If you've already trained over a large set of data, can you un-train it for some of the inputs?
    - dumitrue 14y ago
      That's an interesting (and relevant) question. In fact, if you're using something like stochastic gradient descent to optimize the weights of your network, it might be very hard for the network to escape the general local minimum (or basin of attraction) in which it ended after having trained over a large dataset, even if you present it with the same examples but with the labels flipped (which would be the easiest way to "unlearn"). In theory stochastic gradient descent allows you to escape local minima: the noise in the stochastic error surface will likely be enough for the network to escape whatever minimum it is in. Practically, because weights will tend to have a large magnitude and because most of the times you'll be using a saturating non-linearity (such as a sigmoid), the number of steps required to escape that local minimum might be too big. Presumably, you could use second-order optimization methods to perhaps escape from minima -- because it allows you to make "bigger" steps -- but that comes with its own set of problems (negative curvature being one of them). I encourage you to actually test these hypotheses: train a simple network on something stupid like MNIST, and make it achieve a reasonable error with many passes through the data. Then change the labels of 10-20-50% of your inputs and continue training (with the same learning rate... or not!) to see how long it takes for the network to get to another minimum.
krickle 14y ago
Something I have been curious about for a bit; how do you include textual data as input into a NN?
- dumitrue 14y ago
  Here's a paper describing the main idea behind doing this: http://ronan.collobert.com/pub/matos/2011_nlp_jmlr.pdf http://ronan.collobert.com/pub/matos/2011_nlp_jmlr.pdf In a nutshell, you learn a vector of real-valued parameters for each word in your vocabulary. To train a network on sequences of words, you represent said sequence as a concatenation of the vectors of these words, and feed it as an input to the network. To learn these vectors, you define the problem of "language modeling" as that of discriminating between two sets of sequences: S1 and S2. S1 is the set of sequences that occur in Wikipedia (of which there are many) and S2 is the same as S1, but where you replace a word in each sequence with a randomly chosen word from your vocabulary (which makes it, with very high probability, an invalid sequence of words). Basically, by learning to discriminative between "good" and "bad" English word sequences, you can learn a language model of sorts. The model is represented by those vectors for each word. You can then project those vectors into 2D, as bravura did a while ago, and look at what is close to each other: http://www.cs.toronto.edu/~hinton/turian.png http://www.cs.toronto.edu/~hinton/turian.png
  - marshallp 14y ago
    The new distbelief paper doesn't explain some things. Why are "receptive fields". An earlier paper claimed better than 15% results on imagenet and so does this, what changed. Why have you claimed sgd works on nonconvex well without comparing against actual nonconvex opto algs. You also didn't cite my comments on metaoptimize where some of your ideas seem to come from.
junto 14y ago
With regards to distributed processing of neural networks, how do you split up the neural network onto the various nodes? Do you cut it up into sections and then compute the combined weights from each nodes outputs?
ajays 14y ago
My NN knowledge is minimal, but from what I recall, the claim always was that you don't need more than 1 hidden layer; and too many hidden neurons results in over-fitting. What changed?
- tgflynn 14y ago
  The claim that you don't need more than 1 hidden layer is based on a mathematical theorem that says roughly that a sufficiently large 3 layer net exists that can fit any sufficiently smooth function. One has to look in detail at things like "sufficiently smooth" and "sufficiently large" when applying mathematical theorems to real problems - and that's a step practitioners often seem to neglect when looking for rules of thumb. Also just because a net exists doesn't mean that any given training algorithm is likely to find it. As for overfitting the best way to reduce it is to use more training data and I believe the nets discussed in these papers have been trained on some of the largest training sets ever used. The other problem with deep networks is that they have been considered very difficult to train with backpropagation due to vanishing or exploding gradients. I think the recent major algorithmic developments have mainly involved methods to mitigate these problems.

by bravura 14y ago

I did my postdoctoral research on deep learning, and got into it back when Geoff Hinton's work was just an unpublished tech report. So if anyone has any questions on it, I will try to answer.

e98cuenc 14y ago

Is there any open source ML framework that includes support for DNN? Do you know of any tutorial that may guide the beginner using DNN, I have no idea how to choose the number of hidden layers and activation functions. Thanks!

dumitrue 14y ago
http://deeplearning.net/software/theano/ http://deeplearning.net/software/theano/ is a good place to start. It's open source, in Python, has a few tutorials that lead you towards some rather state-of-the-art methods. There's no secret sauce for how to choose the number of layers and the activation functions (and anyone that tells you otherwise is lying). It's all application-dependent and the best way to do it is by cross-validation. Can answer questions about this topic in conjunction with mr. bravura (I'm working with the code referenced in that paper by Dean et al., here at Google, and did grad school at the same place as bravura did his postdoc).
- e98cuenc 14y ago
  I'm a xoogler, stuff like this makes me wish I was back there :) I want to play a bit with this stuff and have some fun, so thanks for the Theano reference, seems cool! WRT the DNN parameters, is it possible to try all potential possibilities (within reason) and find the best one using only cross-validation, or are there just too many choices and you have to use intuition? (from your comment I don't get if cross-validation is ok to get optimal number of layers etc. or if you have to be "smart") Thanks for replying!
  - dumitrue 14y ago
    For things like the number of layers, the total number of options is relatively small -- usually people try between 1 and 6-7. For most of the other parameters you have to be smarter than that, especially since a lot of them are real-valued, so you can't really explore them all. One of the trends these days is to perform automatic hyper-parameter tuning, especially for cases where a full exploration of hyper-parameters via grid-search would mean a combinatorial explosion of possibilities (and for neural networks you can conceivably explore dozen of hyper-parameters). A friend of mine just got a paper published at NIPS (same conference) on using Bayesian optimization/Gaussian processes for optimizing the hyper-parameters of a model -- http://www.dmi.usherb.ca/~larocheh/publications/gpopt_nips.pdf http://www.dmi.usherb.ca/~larocheh/publications/gpopt_nips.p.... They get better than state of the art results on a couple of benchmarks, which is neat. The code is public -- http://www.cs.toronto.edu/~jasper/software.html http://www.cs.toronto.edu/~jasper/software.html -- and in python, so you could potentially try it out (runs on EC2, too). Btw, Geoff Hinton is teaching an introductory neural nets class on Coursera these days, you should check it out, he's a great teacher. Also, you can always come back to Google, we're doing cool stuff with this :)
seiji 14y ago
Ask me again in two weeks when my online course in advanced machine learning for extreme beginners kicks off. We won't start with deep learning immediately, but it's in the pipeline not too soon after launch.
- e98cuenc 14y ago
  I hope I'm not getting too off-topic, but will this course be taught in coursera or somewhere else? (I want to put a reminder to check this out). Thanks!
  - seiji 14y ago
    It'll be on a new site for teaching complex things in non-complicated ways. The goal is to allow everyone from clever middle school students through retired people to understand the coming changes to the world. There's a huge on-site community focus too. We don't want there to be 100,000 anonymous people just going through the motions. There will be plenty of interaction between course material and community feedback. It's kinda awesome. Topics will be presented in multiple ways (simple and intermediate) so you can have plenty of different views on the same material. The material works as both zero-knowledge intro to the topics as well as quick refreshers if you haven't seen the material in a while (quick -- what's an eigenvector?!). The launch courses will be 1.) real-world applications of probability and statistics (signal extractions), 2.) linear algebra for computer science, and 3.) wildcard (a random assortment of whatever the heck we think is important or entertaining to know). Future courses are: introduction to neural networks, introduction to computational neuroscience, introduction to deep learning, advanced deep learning, how to take over the world with a few dozen GPUs, avenues by which google will become irrelevant, and robotics for fun and evil. This is phase zero of a four phase plan. I'll get some pre-launch material together to shove down HN shortly, then it'll launch a few days later. Hopefully you'll hear about the project again.
    - stephenlee 14y ago
      Great work. It's so cool, can't wait to take part in.

ArbitraryLimits 14y ago

Please tell me whether the term "deep learning" has any meaning at all, and if so, what is it? It sounds like a buzzword to me, like "data scientist."

dumitrue 14y ago
To some extent it's true that "deep learning" is a buzzword, though it is less arbitrary that "data scientist" (which makes no sense whatsoever). In a nutshell, what people mean by "deep learning" is the collection of tips and tricks that make it possible to efficiently train multiple layers of non-linear hidden units (or feature extractors). Sometimes these are multi-layer neural nets, but they don't have to be.
- fudged71 14y ago
  I'm in the middle of a machine learning class and it's all very interesting, but I've got one question for you: How difficult is it to make a neural net "forget"? If you've already trained over a large set of data, can you un-train it for some of the inputs?
  - dumitrue 14y ago
    That's an interesting (and relevant) question. In fact, if you're using something like stochastic gradient descent to optimize the weights of your network, it might be very hard for the network to escape the general local minimum (or basin of attraction) in which it ended after having trained over a large dataset, even if you present it with the same examples but with the labels flipped (which would be the easiest way to "unlearn"). In theory stochastic gradient descent allows you to escape local minima: the noise in the stochastic error surface will likely be enough for the network to escape whatever minimum it is in. Practically, because weights will tend to have a large magnitude and because most of the times you'll be using a saturating non-linearity (such as a sigmoid), the number of steps required to escape that local minimum might be too big. Presumably, you could use second-order optimization methods to perhaps escape from minima -- because it allows you to make "bigger" steps -- but that comes with its own set of problems (negative curvature being one of them). I encourage you to actually test these hypotheses: train a simple network on something stupid like MNIST, and make it achieve a reasonable error with many passes through the data. Then change the labels of 10-20-50% of your inputs and continue training (with the same learning rate... or not!) to see how long it takes for the network to get to another minimum.

krickle 14y ago

Something I have been curious about for a bit; how do you include textual data as input into a NN?

dumitrue 14y ago
Here's a paper describing the main idea behind doing this: http://ronan.collobert.com/pub/matos/2011_nlp_jmlr.pdf http://ronan.collobert.com/pub/matos/2011_nlp_jmlr.pdf In a nutshell, you learn a vector of real-valued parameters for each word in your vocabulary. To train a network on sequences of words, you represent said sequence as a concatenation of the vectors of these words, and feed it as an input to the network. To learn these vectors, you define the problem of "language modeling" as that of discriminating between two sets of sequences: S1 and S2. S1 is the set of sequences that occur in Wikipedia (of which there are many) and S2 is the same as S1, but where you replace a word in each sequence with a randomly chosen word from your vocabulary (which makes it, with very high probability, an invalid sequence of words). Basically, by learning to discriminative between "good" and "bad" English word sequences, you can learn a language model of sorts. The model is represented by those vectors for each word. You can then project those vectors into 2D, as bravura did a while ago, and look at what is close to each other: http://www.cs.toronto.edu/~hinton/turian.png http://www.cs.toronto.edu/~hinton/turian.png
- marshallp 14y ago
  The new distbelief paper doesn't explain some things. Why are "receptive fields". An earlier paper claimed better than 15% results on imagenet and so does this, what changed. Why have you claimed sgd works on nonconvex well without comparing against actual nonconvex opto algs. You also didn't cite my comments on metaoptimize where some of your ideas seem to come from.

junto 14y ago

With regards to distributed processing of neural networks, how do you split up the neural network onto the various nodes? Do you cut it up into sections and then compute the combined weights from each nodes outputs?

ajays 14y ago

My NN knowledge is minimal, but from what I recall, the claim always was that you don't need more than 1 hidden layer; and too many hidden neurons results in over-fitting. What changed?

tgflynn 14y ago
The claim that you don't need more than 1 hidden layer is based on a mathematical theorem that says roughly that a sufficiently large 3 layer net exists that can fit any sufficiently smooth function. One has to look in detail at things like "sufficiently smooth" and "sufficiently large" when applying mathematical theorems to real problems - and that's a step practitioners often seem to neglect when looking for rules of thumb. Also just because a net exists doesn't mean that any given training algorithm is likely to find it. As for overfitting the best way to reduce it is to use more training data and I believe the nets discussed in these papers have been trained on some of the largest training sets ever used. The other problem with deep networks is that they have been considered very difficult to train with backpropagation due to vanishing or exploding gradients. I think the recent major algorithmic developments have mainly involved methods to mitigate these problems.