7 ms·

I'm curious to hear from current researchers: Back when I did some work in neural networks (10-15 years ago), the field had become saturated with 'meta' papers

by siavosh 10y ago

I'm curious to hear from current researchers: Back when I did some work in neural networks (10-15 years ago), the field had become saturated with 'meta' papers on topology generation, and also on optimization techniques, (slightly better gradient descent, pruning techniques etc). Neural networks had become more art than science. At the end of that period, I got the distinct sense that these were signs that the field had stalled in any significant break throughs. Soon after they fell out favor for well known reasons. I'm curious outside of the current hype, if there's a sense of that now with the current field practitioners and saturation of papers.

malux85 10y ago
Topology generation requires the evaluation of many models - For non-trivial datasets/model sizes the evaluation of a model can be hours, days or weeks, even on current GPUs. There's much better tooling in place, and hardware in place( modern GPUs) now to distribute the evaluation over a cluster of GPU machines, so it's becoming practical to push the boundaries further edit: I also forgot to mention, older methods often required the manual specification of heuristics for mutation and crossover rules, this is no longer the case, because the machines can be trained with objective functions to prevent overfitting, optimize execution speed or memory usage and can use techniques like dropout, skip connections, layer / node hyperparameters and so on to direct the topology search
- hellofunk 10y ago
  Is NEAT among the methods you refer to?
  - malux85 10y ago
    Yes, but take it way further. i.e. How can you use a Deep Network to direct NEAT? Can you model the topology generation as a latent space, and what would some of the properties of that space be? Can we integrate the best parts of transfer learning to speed up the generate -> evaluate cycle. That sort of thing
    - sjg007 10y ago
      Sure sounds reasonable. This feels like Markov chain Monte Carlo.
PaulHoule 10y ago
I am an outsider but looking at the last year I think significant improvements are going on right now in both deep networks and neural nets. If you were going into CS grad school for a PhD it is very possible that the bubble might pop before you graduate.
- Florin_Andrei 10y ago
  There is disagreement on whether this is currently a bubble or the next big thing. I believe evidence so far points at the latter.
  - PaulHoule 10y ago
    Those are not contradictory! The story of most academic developments is that somebody develops a new technique, then they solve the problems that are easy to solve with the technique. At some point the problems that can be solved with the technique are solved and nobody has an idea of how to break the technique's limitations. Maybe 20 years after that, somebody gets a new idea. Some examples: perceptrons, symbolic A.I., multilayer perceptrons, renormalization group theory in physics, etc. It can take 10 years to get a PhD so it is not impossible that the field can burn out before then. I don't believe it is burning out in 2 or 3 years.
- sjg007 10y ago
  Well study AI for the sake of AI not because it is currently sexy. When I studied AI it was all about planning systems. The next wave was Bayesian networks and now it is deep learning and reinforcement learning. There will be something else after that... just the other day we had an article on DL and then prolog style learning to generalize the numerous to symbolics. So the field is at a precipice so to speak. DL is in the application phase and then there's a bunch of theoretical work to do. The theory is probably the domain of mathematicians at this point so melding DL with other AI techniques seems the most promising. That also has multiple applications. You can imagine self learning rule based systems as the next big thing.
  - sjg007 10y ago
highd 10y ago
The improvements to gradient descent techniques over vanilla gradient descent as well as ideas like batch normalization and residual nets have made even moderately large networks that were previously impossible to train now possible, let alone computationally tractable. Most of this work was done in the last 5-6 years. Furthermore, training can now be made drastically more robust, with far fewer empirically set parameters. This has made deep neural nets far more useful in practical application.
- siavosh 10y ago
  I don't doubt that significant progess has been made in the last five or so years. Meta papers just make me curious given my past experience: the field had ended up with so many tunable parameters that no one understood, it gave rise to whole new papers to 'learn' some arbitrary set of those parameters by a different method. The dirty secret was you were simply adding more parameters. Thus the real issue was the research field had stalled and hit a temporary dead end.
eva1984 10y ago
Not a theory person, so I didn't quite follow whether or to what extent deep learning has made breakthrough on that front. But at this moment, we are far from done playing with what we already discovered. Even as a black box, deep learning already cracked many traditional tasks to near/above human performance, and enables many application that is simply not possible before. So, I guess we are going to more papers to come in the following years until no low-hanging fruits left.
LeanderK 10y ago
Disclaimer: Interested in NN, but no real knowledge yet. A ML-learning group at my university recently discussed Learning to learn by gradient descent by gradient descent [1], the topic seems related. This indicates to me that at least some progress was made. [1] https://arxiv.org/abs/1606.04474 https://arxiv.org/abs/1606.04474
cr0sh 10y ago
First off, I'm not an expert - right now, I'm an amateur student of the field. I don't have any credentials or anything published. In other words, what I'm going to say probably has little to no merit to it. What I have seen in recent weeks, as I've investigated different news and articles about machine learning, neural networks, and self-driving vehicles, etc - is that it seems like, at least maybe for a certain class of problems, there is a convergence on a generalized pattern, if not a solution, for implementing the network. This pattern or solution seems built off of the Lecun LeNet MNIST convolutional neural network; specifically, the pattern seems to be roughly: (input) -> (1:n - conv layers) -> (flatten) -> (1:n - fully connected layers) -> (output) where: (1:n - conv layers) = (conv layer) -> (loss layer) -> (activation layer) and: (1:n - fully connected layers) = (fully connected layer) -> (loss layer) -> (activation layer) The (loss layer) is optional, but given that the (activation layer) seems to have converged (in most cases I've seen, which is probably not representative) to RELU, the (loss layer) is needed (sometimes) to prevent overfitting (simple dropout can work well, for instance). I don't want to say that this pattern is the "be-all-end-all" of deep learning, but I have found it curious just how many different problems it can be successfully applied to. The RELU operation, while not being differentiable at zero - seems to work regardless (and there are other activation functions similar to RELU that are differentiable - like softplus - if needed). Anyhow - these "building blocks" seem like the basics - the "lego" for deep learning; taking it from art to engineering. As a student, I've been playing with TensorFlow - and recently Keras (a framework for TensorFlow). These tools make it quick and easy to build deep learning neural networks like I described. My gut feeling is that I'm talking from my rear; my level of knowledge isn't that great in this field. So - grain 'o salt and all that. /disclaimer: I am currently enrolled and taking the Udacity Self Driving Car Engineer Nanodegree; in the past I've taken the Udacity CS373 course (2012) and the Standford-sponsored ML Class (2011).
- argonaut 10y ago
  When you say "loss layer" I think you mean output? The notation is a bit mixed up. Those building blocks are precisely the art (it's both art and engineering) that siavosh is referring to. Everyone is mostly using derivatives of the same basic MNIST architecture of LeCun from 20 years ago ('97), because that's what works (no other justification). With tweaks of course (ReLU, batch norm, etc), also because "they just work." It's funny you should mention dropout, because that's falling out of favor (fully-connected layers are also falling out of favor). Why? Because in many cases it doesn't seem to be necessary if you do batch norm. That's art and engineering (basically you try with dropout and batch norm, or with just batch norm, and just go with batch norm if no difference).
deepnotderp 10y ago
Meta-DL is totally possible. Check neuroevolution and NNA search through RL
ma2rten 10y ago
There is no sign of a slowdown as of yet. See Andrew Ng's take on this question: https://www.technologyreview.com/s/603062/ai-winter-isnt-coming/ https://www.technologyreview.com/s/603062/ai-winter-isnt-com... Training neural networks is still an art to a degree, but it is becoming better understood, not less.
nl 10y ago
GANs (and InfoGAN[0] in particular) has shown there are still pretty huge breakthoughs out there. Some other things you may be interested in: Learning to learn by gradient descent by gradient descent[1] Neural Architecture Search with Reinforcement Learning[2] [0] https://arxiv.org/abs/1606.03657 https://arxiv.org/abs/1606.03657, https://arxiv.org/pdf/1701.00160.pdf https://arxiv.org/pdf/1701.00160.pdf [1] https://arxiv.org/abs/1606.04474 https://arxiv.org/abs/1606.04474 [2] https://openreview.net/pdf?id=r1Ue8Hcxg https://openreview.net/pdf?id=r1Ue8Hcxg
niels_olson 10y ago
My sense is that the major issue is not the algorithms. The algorithms are plenty fine today. The major issue is data. Not "more" data. Just data. There are vast swaths of the economy where people simply haven't figured out how to digitize the data yet. They have no idea the gold mines they are sitting on. Imagine everyone who never took multi-dimensional calculus in college (most of humanity). Where are they today? Those folks have data they're not using.

by siavosh 10y ago

malux85 10y ago

Topology generation requires the evaluation of many models - For non-trivial datasets/model sizes the evaluation of a model can be hours, days or weeks, even on current GPUs. There's much better tooling in place, and hardware in place( modern GPUs) now to distribute the evaluation over a cluster of GPU machines, so it's becoming practical to push the boundaries further edit: I also forgot to mention, older methods often required the manual specification of heuristics for mutation and crossover rules, this is no longer the case, because the machines can be trained with objective functions to prevent overfitting, optimize execution speed or memory usage and can use techniques like dropout, skip connections, layer / node hyperparameters and so on to direct the topology search

hellofunk 10y ago
Is NEAT among the methods you refer to?
- malux85 10y ago
  Yes, but take it way further. i.e. How can you use a Deep Network to direct NEAT? Can you model the topology generation as a latent space, and what would some of the properties of that space be? Can we integrate the best parts of transfer learning to speed up the generate -> evaluate cycle. That sort of thing
  - sjg007 10y ago
    Sure sounds reasonable. This feels like Markov chain Monte Carlo.

PaulHoule 10y ago

I am an outsider but looking at the last year I think significant improvements are going on right now in both deep networks and neural nets. If you were going into CS grad school for a PhD it is very possible that the bubble might pop before you graduate.

Florin_Andrei 10y ago
There is disagreement on whether this is currently a bubble or the next big thing. I believe evidence so far points at the latter.
- PaulHoule 10y ago
  Those are not contradictory! The story of most academic developments is that somebody develops a new technique, then they solve the problems that are easy to solve with the technique. At some point the problems that can be solved with the technique are solved and nobody has an idea of how to break the technique's limitations. Maybe 20 years after that, somebody gets a new idea. Some examples: perceptrons, symbolic A.I., multilayer perceptrons, renormalization group theory in physics, etc. It can take 10 years to get a PhD so it is not impossible that the field can burn out before then. I don't believe it is burning out in 2 or 3 years.
sjg007 10y ago
Well study AI for the sake of AI not because it is currently sexy. When I studied AI it was all about planning systems. The next wave was Bayesian networks and now it is deep learning and reinforcement learning. There will be something else after that... just the other day we had an article on DL and then prolog style learning to generalize the numerous to symbolics. So the field is at a precipice so to speak. DL is in the application phase and then there's a bunch of theoretical work to do. The theory is probably the domain of mathematicians at this point so melding DL with other AI techniques seems the most promising. That also has multiple applications. You can imagine self learning rule based systems as the next big thing.
- sjg007 10y ago

highd 10y ago

The improvements to gradient descent techniques over vanilla gradient descent as well as ideas like batch normalization and residual nets have made even moderately large networks that were previously impossible to train now possible, let alone computationally tractable. Most of this work was done in the last 5-6 years. Furthermore, training can now be made drastically more robust, with far fewer empirically set parameters. This has made deep neural nets far more useful in practical application.

siavosh 10y ago
I don't doubt that significant progess has been made in the last five or so years. Meta papers just make me curious given my past experience: the field had ended up with so many tunable parameters that no one understood, it gave rise to whole new papers to 'learn' some arbitrary set of those parameters by a different method. The dirty secret was you were simply adding more parameters. Thus the real issue was the research field had stalled and hit a temporary dead end.

eva1984 10y ago

Not a theory person, so I didn't quite follow whether or to what extent deep learning has made breakthrough on that front. But at this moment, we are far from done playing with what we already discovered. Even as a black box, deep learning already cracked many traditional tasks to near/above human performance, and enables many application that is simply not possible before. So, I guess we are going to more papers to come in the following years until no low-hanging fruits left.

LeanderK 10y ago

Disclaimer: Interested in NN, but no real knowledge yet. A ML-learning group at my university recently discussed Learning to learn by gradient descent by gradient descent [1], the topic seems related. This indicates to me that at least some progress was made. [1] https://arxiv.org/abs/1606.04474 https://arxiv.org/abs/1606.04474

cr0sh 10y ago

First off, I'm not an expert - right now, I'm an amateur student of the field. I don't have any credentials or anything published. In other words, what I'm going to say probably has little to no merit to it. What I have seen in recent weeks, as I've investigated different news and articles about machine learning, neural networks, and self-driving vehicles, etc - is that it seems like, at least maybe for a certain class of problems, there is a convergence on a generalized pattern, if not a solution, for implementing the network. This pattern or solution seems built off of the Lecun LeNet MNIST convolutional neural network; specifically, the pattern seems to be roughly: (input) -> (1:n - conv layers) -> (flatten) -> (1:n - fully connected layers) -> (output) where: (1:n - conv layers) = (conv layer) -> (loss layer) -> (activation layer) and: (1:n - fully connected layers) = (fully connected layer) -> (loss layer) -> (activation layer) The (loss layer) is optional, but given that the (activation layer) seems to have converged (in most cases I've seen, which is probably not representative) to RELU, the (loss layer) is needed (sometimes) to prevent overfitting (simple dropout can work well, for instance). I don't want to say that this pattern is the "be-all-end-all" of deep learning, but I have found it curious just how many different problems it can be successfully applied to. The RELU operation, while not being differentiable at zero - seems to work regardless (and there are other activation functions similar to RELU that are differentiable - like softplus - if needed). Anyhow - these "building blocks" seem like the basics - the "lego" for deep learning; taking it from art to engineering. As a student, I've been playing with TensorFlow - and recently Keras (a framework for TensorFlow). These tools make it quick and easy to build deep learning neural networks like I described. My gut feeling is that I'm talking from my rear; my level of knowledge isn't that great in this field. So - grain 'o salt and all that. /disclaimer: I am currently enrolled and taking the Udacity Self Driving Car Engineer Nanodegree; in the past I've taken the Udacity CS373 course (2012) and the Standford-sponsored ML Class (2011).

argonaut 10y ago
When you say "loss layer" I think you mean output? The notation is a bit mixed up. Those building blocks are precisely the art (it's both art and engineering) that siavosh is referring to. Everyone is mostly using derivatives of the same basic MNIST architecture of LeCun from 20 years ago ('97), because that's what works (no other justification). With tweaks of course (ReLU, batch norm, etc), also because "they just work." It's funny you should mention dropout, because that's falling out of favor (fully-connected layers are also falling out of favor). Why? Because in many cases it doesn't seem to be necessary if you do batch norm. That's art and engineering (basically you try with dropout and batch norm, or with just batch norm, and just go with batch norm if no difference).

deepnotderp 10y ago

Meta-DL is totally possible. Check neuroevolution and NNA search through RL

ma2rten 10y ago

There is no sign of a slowdown as of yet. See Andrew Ng's take on this question: https://www.technologyreview.com/s/603062/ai-winter-isnt-coming/ https://www.technologyreview.com/s/603062/ai-winter-isnt-com... Training neural networks is still an art to a degree, but it is becoming better understood, not less.

nl 10y ago

GANs (and InfoGAN[0] in particular) has shown there are still pretty huge breakthoughs out there. Some other things you may be interested in: Learning to learn by gradient descent by gradient descent[1] Neural Architecture Search with Reinforcement Learning[2] [0] https://arxiv.org/abs/1606.03657 https://arxiv.org/abs/1606.03657, https://arxiv.org/pdf/1701.00160.pdf https://arxiv.org/pdf/1701.00160.pdf [1] https://arxiv.org/abs/1606.04474 https://arxiv.org/abs/1606.04474 [2] https://openreview.net/pdf?id=r1Ue8Hcxg https://openreview.net/pdf?id=r1Ue8Hcxg

niels_olson 10y ago

My sense is that the major issue is not the algorithms. The algorithms are plenty fine today. The major issue is data. Not "more" data. Just data. There are vast swaths of the economy where people simply haven't figured out how to digitize the data yet. They have no idea the gold mines they are sitting on. Imagine everyone who never took multi-dimensional calculus in college (most of humanity). Where are they today? Those folks have data they're not using.