6 ms·

The Neural Network Zoo

358 points by hgarg 10y ago

cs702 10y ago
Good job. Alas, the proliferation of different kinds of neural net architectures that work, over the past few years, is a sign that we lack a decent unifying theoretical framework that can explain, from fundamental principles, what works, what doesn't, and why. We're not there yet.
- SomeStupidPoint 10y ago
  > the proliferation of different kinds of neural net architectures that work, over the past few years, is a sign that we lack a decent unifying theoretical framework that can explain, from fundamental principles, what works, what doesn't, and why Could you elaborate on this? I ask, because it's not obvious to me that this is true. Physics, for instance, has had proliferations of new "particles" both because they lacked a solid understanding of a phenomena (but knew how to tweak the setup to vary the result) and because they had a new, solid theoretical framework that was able to accurately predict whole classes of new particles. (There's a bunch of unstable particles related to the electroweak interaction that are from this, for instance.) As someone who has reached "journeyman" at the topic, it's sort of hard to tell if people are just tweaking the setup to get new experimental results they can use to brute force theory building or whether they're setting out on a specific research path on solid theoretical grounds. (I expect that it's a mix of the two, with different shops being different blends, to be honest.)
  - fazzone 10y ago
    Well, a lot of the papers published in the field present results like "We designed a neural net to perform <task> and achieved X% accuracy." The design of the net is novel and interesting enough to merit its own publication. If there was some sort of theoretical framework, results like that would not be interesting, because presumably the theory would explain which NN architectures are good at different tasks and why. I think that we will get there eventually, but right now I don't we have enough data for patterns to emerge and hint at some sort of Theory.
    - SomeStupidPoint 10y ago
      But couldn't that just be, to use the language of my analogy, papers being published which confirm particle existence? I mean, if physics publishes papers when they find things they expected to find (at least, the first instance of each kind of thing and thereafter, any novel improvement in their production), why wouldn't machine learning theorists? That's precisely what I can't tell: is a paper that's "We found that architecture X performed task Y with score Z" the same as "We found particle X at energy level Y and have no idea what it is" or "We found particle X at energy level Y just as we were expecting"? And a lot of the papers are "We changed architecture X to now have feature Y and got the expected improvement Z", which I doubly can't tell how expected the improvement was and how systemically improvements are being designed and implemented.
      - bbctol 10y ago
        I think the issue here is that we're not discovering new architectures as one discovers particles, we're creating them. The creative space is infinite, and people are making subtle tweaks to neural network systems all the time. It's not science, it's engineering. Right now, we're in the early stages of engineering, like architecture before modern physics: we know some things that work, and we have some good intuitions about why, but there's little solid foundation to tell us how to proceed. We take some loose inspiration from nature (replace tanh functions with rectifiers to mimic action potentials in the brain, build convolutional networks similar to the retina) and find that it's more effective, sometimes, and sometimes less. We also just try a lot of stuff in hopes that something will stick. It's not as if there are some real, true neural networks out there, waiting like particles to be discovered: everything in the neural network zoo was built by hand, maybe inspired by nature, and saved because it works well, or is at least interesting; other architectures are forgotten. What we'd like is engineering principles that we can understand, so trying to make a neural network better at function x is just a matter of adding more units here or editing a function there, not venturing out into the dark again. (Such a reductive set of explanations may not exist for cognition, which really worries people who liked computers for their predictability.)
        10y ago
  - spott 10y ago
    Think about neural nets as emergent: simple nodes give rise to complicated dynamics. The dynamics are determined by the overall geometry of the system. Theoretically, we should be able to look at a bunch of different geometries and explain what works and what doesn't, just from the original rules and the geometry. The fact that we can't, and usually experimentation is necessary to get a good idea about the strengths and weaknesses of different geometries, is a signal that we don't really understand the system. I'm not aware of any theory of the type: give me an ordered graph with typed nodes and I'll give you the types of data this is good at modeling, how quickly it will converge, if you need dropout or layer-skip or something else in order to make it converge well, etc. In your physics analogy: we can write out various properties of composite particles purely from theory: i.e. Mesons decay pathways and allowed compositions, Baryons decay pathways and allowed compositions, etc, just knowing about the core 17 standard model particles. We know about a few core nodes of neural nets, but can't use that to predict the behavior of the whole net.
- nkozyra 10y ago
  I assume you mean what works for what problem. I think the problem is these are moving so fast that what didn't work yesterday has been coerced into being the best option for, say, image classification today. Hell, ANN itself was widely the last choice for ML problems until a couple of years back and now it seems like the first choice go-to.
- Houshalter 10y ago
  Look at computer science. They have invented so many different algorithms that work! This is a sign that computer science lacks a decent unifying theoretical framework. Having different algorithms for different purposes is fine. For instance, autoencoders can do unsupervised learning, GANs can learn generative models that sample from the entire distribution, recurrent neural networks can handle time series, etc. Also while there are many different types of networks, research has shown which ones work and which ones don't. Few people pretrain autoencoders or bother with RBMs anymore, for instance. And I think we have good theoretical reasons why they aren't as good. But to continue the analogy to computer science, imagine all the different kinds of sorting algorithms. They will each work better or worse based on how the data you are sorting is arranged. If it's already sorted in reverse, that's a lot different than if it's sorted completely randomly, or if it was sorted and then big chunks were randomly rearranged. There's no way to prove that one sorting algorithm will always do better than another, because there's always special cases where they do do better. The same is true of neural networks, it's impossible to formally prove they will work, because it depends on how real world problems are distributed.
  - AndrewKemendo 10y ago
    This is a sign that computer science lacks a decent unifying theoretical framework. Yea I think that's probably true as well. Do you consider the VonNeumann architecture to the the unifying theoretical framework? If not then what?
    - LeanderK 10y ago
      I don't think there is a lack of a unifying theoretical framework. Both the turing machine and the lambda calculus are well understood and the foundations of computer science (and are isomorphic to each other). I think the problem is more that a gap exists between the theoretical framework and practical programming. For example have fun trying to formally verify Javascript code! Interesting side-note: we are currently unable to prove that there is nothing more powerful than a turing machine, i don't know if its even provable (that smells like something undecidable).
- codekansas 10y ago
  On the contrary, the concept of a "neural network" is beautifully cohesive. Many of the architectural differences are only iterative; the fundamental units are very similar. Like how there's a lot of diversity in biology, but the fundamental units are shared.
  - deleted 10y ago
    [deleted]
- drabiega 10y ago
  I think a lot of the time we do have a good idea why a certain neural network is better or worse at various tasks. The core problem is that the ability of the network to work is is a function of the nature of the domain as much if not more so than the networks themselves and it is very hard to get an understanding of the shape of the problem space aside from post-hoc insights gained from what type of networks work well and which don't in that particular domain.
- bra-ket 10y ago
  there is a decent unifying theoretical framework behind this: https://en.wikipedia.org/wiki/Infinite_monkey_theorem https://en.wikipedia.org/wiki/Infinite_monkey_theorem
- aab0 10y ago
  We have to hardwire architectures because we can't learn architectures yet, or to put it another way, backpropagation doesn't yet work on hyperparameters as well as on parameters. Hyperparameters should be learnable as in theory there's nothing special about them (it's models all the way down!) - a hyperparameter is merely a parameter we don't yet know how to learn - and this has been demonstrated: http://jmlr.org/proceedings/papers/v37/maclaurin15.pdf http://jmlr.org/proceedings/papers/v37/maclaurin15.pdf "Gradient-based Hyperparameter Optimization through Reversible Learning" "Tuning hyperparameters of learning algorithms is hard because gradients are usually unavailable. We compute exact gradients of cross-validation performance with respect to all hyperparameters by chaining derivatives backwards through the entire training procedure. These gradients allow us to optimize thousands of hyperparameters, including step-size and momentum schedules, weight initialization distributions, richly parameterized regularization schemes, and neural network architectures. We compute hyperparameter gradients by exactly reversing the dynamics of stochastic gradient descent with momentum." But it's not feasible yet. Once it is, you can imagine collapsing the whole neural net zoo: you merely specify the input/output type/dimension and then it starts gradient-ascent over all the possible models as tweaked by internal hyperparameters.
gugagore 10y ago
The explanation about MC is too wrong to be useful. What on earth do the "nodes" (states) in a Markov chain have to do with the "nodes" (neurons) of a neural network?

358 points by hgarg 10y ago

cs702 10y ago

Good job. Alas, the proliferation of different kinds of neural net architectures that work, over the past few years, is a sign that we lack a decent unifying theoretical framework that can explain, from fundamental principles, what works, what doesn't, and why. We're not there yet.

SomeStupidPoint 10y ago
> the proliferation of different kinds of neural net architectures that work, over the past few years, is a sign that we lack a decent unifying theoretical framework that can explain, from fundamental principles, what works, what doesn't, and why Could you elaborate on this? I ask, because it's not obvious to me that this is true. Physics, for instance, has had proliferations of new "particles" both because they lacked a solid understanding of a phenomena (but knew how to tweak the setup to vary the result) and because they had a new, solid theoretical framework that was able to accurately predict whole classes of new particles. (There's a bunch of unstable particles related to the electroweak interaction that are from this, for instance.) As someone who has reached "journeyman" at the topic, it's sort of hard to tell if people are just tweaking the setup to get new experimental results they can use to brute force theory building or whether they're setting out on a specific research path on solid theoretical grounds. (I expect that it's a mix of the two, with different shops being different blends, to be honest.)
- fazzone 10y ago
  Well, a lot of the papers published in the field present results like "We designed a neural net to perform <task> and achieved X% accuracy." The design of the net is novel and interesting enough to merit its own publication. If there was some sort of theoretical framework, results like that would not be interesting, because presumably the theory would explain which NN architectures are good at different tasks and why. I think that we will get there eventually, but right now I don't we have enough data for patterns to emerge and hint at some sort of Theory.
  - SomeStupidPoint 10y ago
    But couldn't that just be, to use the language of my analogy, papers being published which confirm particle existence? I mean, if physics publishes papers when they find things they expected to find (at least, the first instance of each kind of thing and thereafter, any novel improvement in their production), why wouldn't machine learning theorists? That's precisely what I can't tell: is a paper that's "We found that architecture X performed task Y with score Z" the same as "We found particle X at energy level Y and have no idea what it is" or "We found particle X at energy level Y just as we were expecting"? And a lot of the papers are "We changed architecture X to now have feature Y and got the expected improvement Z", which I doubly can't tell how expected the improvement was and how systemically improvements are being designed and implemented.
    - bbctol 10y ago
      I think the issue here is that we're not discovering new architectures as one discovers particles, we're creating them. The creative space is infinite, and people are making subtle tweaks to neural network systems all the time. It's not science, it's engineering. Right now, we're in the early stages of engineering, like architecture before modern physics: we know some things that work, and we have some good intuitions about why, but there's little solid foundation to tell us how to proceed. We take some loose inspiration from nature (replace tanh functions with rectifiers to mimic action potentials in the brain, build convolutional networks similar to the retina) and find that it's more effective, sometimes, and sometimes less. We also just try a lot of stuff in hopes that something will stick. It's not as if there are some real, true neural networks out there, waiting like particles to be discovered: everything in the neural network zoo was built by hand, maybe inspired by nature, and saved because it works well, or is at least interesting; other architectures are forgotten. What we'd like is engineering principles that we can understand, so trying to make a neural network better at function x is just a matter of adding more units here or editing a function there, not venturing out into the dark again. (Such a reductive set of explanations may not exist for cognition, which really worries people who liked computers for their predictability.)
      - 10y ago
- spott 10y ago
  Think about neural nets as emergent: simple nodes give rise to complicated dynamics. The dynamics are determined by the overall geometry of the system. Theoretically, we should be able to look at a bunch of different geometries and explain what works and what doesn't, just from the original rules and the geometry. The fact that we can't, and usually experimentation is necessary to get a good idea about the strengths and weaknesses of different geometries, is a signal that we don't really understand the system. I'm not aware of any theory of the type: give me an ordered graph with typed nodes and I'll give you the types of data this is good at modeling, how quickly it will converge, if you need dropout or layer-skip or something else in order to make it converge well, etc. In your physics analogy: we can write out various properties of composite particles purely from theory: i.e. Mesons decay pathways and allowed compositions, Baryons decay pathways and allowed compositions, etc, just knowing about the core 17 standard model particles. We know about a few core nodes of neural nets, but can't use that to predict the behavior of the whole net.
nkozyra 10y ago
I assume you mean what works for what problem. I think the problem is these are moving so fast that what didn't work yesterday has been coerced into being the best option for, say, image classification today. Hell, ANN itself was widely the last choice for ML problems until a couple of years back and now it seems like the first choice go-to.
Houshalter 10y ago
Look at computer science. They have invented so many different algorithms that work! This is a sign that computer science lacks a decent unifying theoretical framework. Having different algorithms for different purposes is fine. For instance, autoencoders can do unsupervised learning, GANs can learn generative models that sample from the entire distribution, recurrent neural networks can handle time series, etc. Also while there are many different types of networks, research has shown which ones work and which ones don't. Few people pretrain autoencoders or bother with RBMs anymore, for instance. And I think we have good theoretical reasons why they aren't as good. But to continue the analogy to computer science, imagine all the different kinds of sorting algorithms. They will each work better or worse based on how the data you are sorting is arranged. If it's already sorted in reverse, that's a lot different than if it's sorted completely randomly, or if it was sorted and then big chunks were randomly rearranged. There's no way to prove that one sorting algorithm will always do better than another, because there's always special cases where they do do better. The same is true of neural networks, it's impossible to formally prove they will work, because it depends on how real world problems are distributed.
- AndrewKemendo 10y ago
  This is a sign that computer science lacks a decent unifying theoretical framework. Yea I think that's probably true as well. Do you consider the VonNeumann architecture to the the unifying theoretical framework? If not then what?
  - LeanderK 10y ago
    I don't think there is a lack of a unifying theoretical framework. Both the turing machine and the lambda calculus are well understood and the foundations of computer science (and are isomorphic to each other). I think the problem is more that a gap exists between the theoretical framework and practical programming. For example have fun trying to formally verify Javascript code! Interesting side-note: we are currently unable to prove that there is nothing more powerful than a turing machine, i don't know if its even provable (that smells like something undecidable).
codekansas 10y ago
On the contrary, the concept of a "neural network" is beautifully cohesive. Many of the architectural differences are only iterative; the fundamental units are very similar. Like how there's a lot of diversity in biology, but the fundamental units are shared.
- deleted 10y ago
  [deleted]
drabiega 10y ago
I think a lot of the time we do have a good idea why a certain neural network is better or worse at various tasks. The core problem is that the ability of the network to work is is a function of the nature of the domain as much if not more so than the networks themselves and it is very hard to get an understanding of the shape of the problem space aside from post-hoc insights gained from what type of networks work well and which don't in that particular domain.
bra-ket 10y ago
there is a decent unifying theoretical framework behind this: https://en.wikipedia.org/wiki/Infinite_monkey_theorem https://en.wikipedia.org/wiki/Infinite_monkey_theorem
aab0 10y ago
We have to hardwire architectures because we can't learn architectures yet, or to put it another way, backpropagation doesn't yet work on hyperparameters as well as on parameters. Hyperparameters should be learnable as in theory there's nothing special about them (it's models all the way down!) - a hyperparameter is merely a parameter we don't yet know how to learn - and this has been demonstrated: http://jmlr.org/proceedings/papers/v37/maclaurin15.pdf http://jmlr.org/proceedings/papers/v37/maclaurin15.pdf "Gradient-based Hyperparameter Optimization through Reversible Learning" "Tuning hyperparameters of learning algorithms is hard because gradients are usually unavailable. We compute exact gradients of cross-validation performance with respect to all hyperparameters by chaining derivatives backwards through the entire training procedure. These gradients allow us to optimize thousands of hyperparameters, including step-size and momentum schedules, weight initialization distributions, richly parameterized regularization schemes, and neural network architectures. We compute hyperparameter gradients by exactly reversing the dynamics of stochastic gradient descent with momentum." But it's not feasible yet. Once it is, you can imagine collapsing the whole neural net zoo: you merely specify the input/output type/dimension and then it starts gradient-ascent over all the possible models as tweaked by internal hyperparameters.

gugagore 10y ago

The explanation about MC is too wrong to be useful. What on earth do the "nodes" (states) in a Markov chain have to do with the "nodes" (neurons) of a neural network?