7 ms·

Neural Architecture Search with Reinforcement Learning

116 points by saycheese 10y ago

saycheese 10y ago
Ran across this research reading this article, "AI Software Learns to Make AI Software" - which is already posted here: https://news.ycombinator.com/item?id=13436195 https://news.ycombinator.com/item?id=13436195
gallerdude 10y ago
I think this is the way that Neural Networks achieve some modicum of generality - chaining them together. Let's say you have a robot that you want to grab a can of beer off the counter. You say "grab that beer" and point to it. The first neural network interprets the speech and visual input. A second neural network chooses the proper neural nets to continue the task based on the information interpreted from the first net - it picks one for walking and one for grabbing.
- randcraw 10y ago
  Of course compound task federation like this is not exactly new. Blackboard systems did the very same thing using expert systems back in the 1970s-80s. The problems then remain today: complex and compound tasks don't decompose neatly into well-defined constituent subtasks. Sub-task recombination rapidly devolves into a rat's nest of selecting from competing subtask components that don't have clearly distinct semantics, that are not free of contextual dependencies, nor do they plug-in-and-play independently.
  - ced 10y ago
    Do you have any interesting material on this? Where's a good in-depth analysis of blackboard systems' pros and cons?
    - randcraw 10y ago
      I'd start here: https://en.wikipedia.org/wiki/Blackboard_system https://en.wikipedia.org/wiki/Blackboard_system I'm not up-to-date, but I haven't heard of new work on blackboards or federated rule-based systems for maybe 20 years now, after expert systems grew increasingly probabilistic (and procrustean), and BBSes based on binary rules showed little sign of escaping the classic brittleness of RBSes. The wiki article mentions 'Bayesian Blackboard' systems. Maybe they had greater success?
    - joshmarlow 10y ago
      BBS systems also interest me quite a bit. Check out https://en.wikipedia.org/wiki/LIDA_(cognitive_architecture) https://en.wikipedia.org/wiki/LIDA_(cognitive_architecture). It's based on a Global Workspace model (conceptually similar to a Black Board System). You can find papers on LIDA online (including one or two from the references on the wiki page). If you find anything particularly interesting, feel free to share!
- westoncb 10y ago
  So maybe now we need an NN that figures out the best way of wiring up various pre-existing NNs?
  - halflings 10y ago
    Isn't that exactly what the paper is about?
    - westoncb 10y ago
      It's sort of related. In the paper they are still learning parameters for a single new network, as opposed to training a new network to be good at linking up other networks.
ericjang 10y ago
The primary author is a Google Brain Resident (https://research.google.com/teams/brain/residency/ https://research.google.com/teams/brain/residency/). The Brain Residency is a really great program for starting a career in Deep Learning and ML research, and I'm really impressed by how quickly these new researchers churn out great work like this. disclosure: I work at Google Brain
- cynicaldevil 10y ago
  Wow this feels tailor made for people like me. I want to do further studies in deep learning after I graduate, but I'm not really keen on spending 4/5 years of my life on it. I'll be keeping an eye on this....
- cr0sh 10y ago
  I really would like to, someday, do the residency, but at the same time, I don't know if I can. Right now, I'm doing the Udacity Self-Driving Car Engineer Nanodegree. I won't rehash that or my other knowledge here... My issue, though, is my age, coupled with the fact that I don't have a real degree (I've got this old associates degree from a tech trade school that is almost worthless). After I finish the Udacity thing, I have this idea of pursuing an online BA, then an MA - likely in CompSci. But, we are likely looking at anywhere from 4 years or more (likely more) to do both things - and not a small amount of money. After that point, I'll be close to 50 years old. There's a very good and likely chance that the residency won't be around at that point; or even the possibility that technology around ML may have radically changed the world to the point where it would be difficult (or even meaningless) to try to "catch up"... Still - I'm not going to let that possibility stop or restrain me; that said, is it viable for me to even think about this kind of thing - doing such a residency? Would I even have a chance of being considered, vs someone younger? Furthermore - how does one do a residency of this nature, while still paying bills (I live in Phoenix, AZ - I own a house, plus have other bills)? These are just questions I have - not really concerns, as I am no where near the point I need to be to do the residency - and there isn't any guarantee that I will be. I'll just continue to enjoy to journey, rather than worrying about a specific end-goal.
  - deleted 10y ago
deepnotderp 10y ago
This is pretty old, and neural nets can train neural nets too (better than humans as usual). Check learning to learn w/ gradient descent by gradient descent
- westoncb 10y ago
  Err.. this paper cites the one you're referring to, so I do not think it's older (also, the dates they were published).
- deepnotderp 10y ago
  (Learning to learn is older, I just thought it would be cool)
jorgemf 10y ago
I would love to see a Neural Architecture Search that create Neural Architecture Search as a future research project. Meta-meta-learning. I like the idea of improving the network which creates other networks. Also the size of the network can be used as part of the evaluation in order to minimize the networks and maximize the accuracy.
- deleted 10y ago
  [deleted]
- westoncb 10y ago
  I would guess the computational complexity involved in training the top level network would make the project infeasible.
  - jorgemf 10y ago
    You can relax the problem in order to reduce complexity and calculations. For example: cut the training of slow learning networks, have a database of already trained networks, decrease the number of epochs and examples, etc. Or even create another network which predicts the convergence of a network and use it as heuristic. If you also take into account the size of the network and force the learning to minimize them, then you can train them faster. We cannot do it at home, but for sure Google can try it in their servers.
Smerity 10y ago
Extreme paper tldr - Humans usually construct neural network components and the graph of how they fit together by hand. This work sets up a "controller" neural network that constructs two core components in many neural networks, an RNN and a CNN, through reinforcement learning. This is an intensive and slow process, requiring 400 CPUs and 800 GPUs for the RNN and CNN respectively, but achieves better than or near state of the art results for language modeling and vision classification respectively. This paper is currently under review for ICLR 2017 and is one of the papers I was most excited about. I previously wrote an article, "In deep learning, architecture engineering is the new feature engineering"[1], which discusses the (often ignored) fact that much of the work in modern deep learning papers is just assembling components in different combinations. This is one of the first works that I feel provides a viable answer to my complaint/concern. The paper itself tackles two problems - first, that optimizing architecture is usually black magic directed poorly by humans, and second, humans rarely spend their time tailoring towards a specific task, instead seeking generality. Zoph and Le do this by having one neural network generate the architecture for a later one through a large series of experiments. They perform experiments in both vision (classification) and text (language modeling), replacing the convolutional neural network component and the recurrent neural network component respectively. First is that many of the choices regarding constructing the neural network architecture are somewhat arbitrary and only hit upon experimentally by the practitioners themselves. Andrej Karpathy noted in one of his lectures (paraphrased) "Start with an architecture that works, then modify from there" - mainly as there's a lot of "black magic" in these architectures that has only been discovered by spilling blood to the experimental god of a hundred GPUs and/or "graduate student descent" (i.e. where you lock a poor grad in a room for an indeterminate period of time and tell them to do better on task X). Being able to turn to a neural network to run this painful search for you instead is a good idea - assuming you have the large number of GPUs or CPUs necessary. In the paper they use 400 CPUs for the language modeling search and 800 GPUs for the CNN classification search! The other is whether we should generalize or specialize these architectures. There are many variants of architectures that are not built for or tested against each possible new task. For example, within recurrent neural networks (RNNs) we have the RNN/GRU/LSTM/QRNN/RHN/... and a million minor minor variants between them, each of which perform slightly different depending on the task. While we would like to imagine the architectures that humans make would get progressively closer to "the perfect generic RNN cell" over time, it makes sense that certain cells are, could, or should be optimized to a specific task. Seeking generality isn't always the correct answer. Humans want to seek generality as we don't have the time to tailor to each specific task - but what if we could? Maybe in that situation Occam's razor is actually an impediment to our thinking. While this is early days, and hugely resource intensive, it is likely to get more feasible over time either as we get more computing power or become smarter regarding how we use it. As a researcher in neural networks, I don't consider this a threat, but instead a useful tool, likely in the same way that a compiler likely only helped assembly programmers. If people are interested, I can write an article covering many of the details of this paper like I did for Google's Neural Machine Translation architecture[2]. In that article I try to step through how these systems work from the ground up, and the reasoning behind many of the decisions in the paper, hopefully in an understandable manner for a general audience. P.S. Merity et al. is one of the numbers they beat in the language modeling section, so you may read this entire post in a bitter tone if you'd like ;) P.P.S. This paper has been out since November 2016 or earlier - I think it was a recent MIT Tech Review article that might have resurfaced it? (oops: wrote Wired initially, meant MIT Tech Review - thanks @saycheese) [1]: http://smerity.com/articles/2016/architectures_are_the_new_feature_engineering.html http://smerity.com/articles/2016/architectures_are_the_new_f... [2]: http://smerity.com/articles/2016/google_nmt_arch.html http://smerity.com/articles/2016/google_nmt_arch.html
jordansmithnz 10y ago
Wait... if the neural net can design other neural nets, can it be taught to design itself?
- mastazi 10y ago
  Ironically, this is one of the oldest ideas in computer science https://en.wikipedia.org/wiki/Von_Neumann_universal_constructor https://en.wikipedia.org/wiki/Von_Neumann_universal_construc...
  - Houshalter 10y ago
    Von Neumanns constructor is about self replication. Not quite an AI that can self improve. I think the first person to put the idea forward was I. J. Good in 1962. He speculated that someday AIs would be good enough to do AI research better than their human masters. Then they would start making even better AIs. Which would make even better AIs, and so on. Leading to what he called an "intelligence explosion". He thought it would be "the last invention that man need ever make." http://web.archive.org/web/20160428183531/http://le-cretin-transnational.ch/wp-content/uploads/2014/04/Good-Speculations-Concerning-the-First-Ultraintelligent-Machine.pdf http://web.archive.org/web/20160428183531/http://le-cretin-t...
    - mastazi 10y ago
      Parent's comment was about self-replication, I was addressing that.
- EvgeniyZh 10y ago
  Can it be taught to design a better version of itself?

116 points by saycheese 10y ago

Ran across this research reading this article, "AI Software Learns to Make AI Software" - which is already posted here: https://news.ycombinator.com/item?id=13436195 https://news.ycombinator.com/item?id=13436195

gallerdude 10y ago

I think this is the way that Neural Networks achieve some modicum of generality - chaining them together. Let's say you have a robot that you want to grab a can of beer off the counter. You say "grab that beer" and point to it. The first neural network interprets the speech and visual input. A second neural network chooses the proper neural nets to continue the task based on the information interpreted from the first net - it picks one for walking and one for grabbing.

randcraw 10y ago
Of course compound task federation like this is not exactly new. Blackboard systems did the very same thing using expert systems back in the 1970s-80s. The problems then remain today: complex and compound tasks don't decompose neatly into well-defined constituent subtasks. Sub-task recombination rapidly devolves into a rat's nest of selecting from competing subtask components that don't have clearly distinct semantics, that are not free of contextual dependencies, nor do they plug-in-and-play independently.
- ced 10y ago
  Do you have any interesting material on this? Where's a good in-depth analysis of blackboard systems' pros and cons?
  - randcraw 10y ago
    I'd start here: https://en.wikipedia.org/wiki/Blackboard_system https://en.wikipedia.org/wiki/Blackboard_system I'm not up-to-date, but I haven't heard of new work on blackboards or federated rule-based systems for maybe 20 years now, after expert systems grew increasingly probabilistic (and procrustean), and BBSes based on binary rules showed little sign of escaping the classic brittleness of RBSes. The wiki article mentions 'Bayesian Blackboard' systems. Maybe they had greater success?
  - joshmarlow 10y ago
    BBS systems also interest me quite a bit. Check out https://en.wikipedia.org/wiki/LIDA_(cognitive_architecture) https://en.wikipedia.org/wiki/LIDA_(cognitive_architecture). It's based on a Global Workspace model (conceptually similar to a Black Board System). You can find papers on LIDA online (including one or two from the references on the wiki page). If you find anything particularly interesting, feel free to share!
westoncb 10y ago
So maybe now we need an NN that figures out the best way of wiring up various pre-existing NNs?
- halflings 10y ago
  Isn't that exactly what the paper is about?
  - westoncb 10y ago
    It's sort of related. In the paper they are still learning parameters for a single new network, as opposed to training a new network to be good at linking up other networks.

ericjang 10y ago

The primary author is a Google Brain Resident (https://research.google.com/teams/brain/residency/ https://research.google.com/teams/brain/residency/). The Brain Residency is a really great program for starting a career in Deep Learning and ML research, and I'm really impressed by how quickly these new researchers churn out great work like this. disclosure: I work at Google Brain

cynicaldevil 10y ago
Wow this feels tailor made for people like me. I want to do further studies in deep learning after I graduate, but I'm not really keen on spending 4/5 years of my life on it. I'll be keeping an eye on this....
cr0sh 10y ago
I really would like to, someday, do the residency, but at the same time, I don't know if I can. Right now, I'm doing the Udacity Self-Driving Car Engineer Nanodegree. I won't rehash that or my other knowledge here... My issue, though, is my age, coupled with the fact that I don't have a real degree (I've got this old associates degree from a tech trade school that is almost worthless). After I finish the Udacity thing, I have this idea of pursuing an online BA, then an MA - likely in CompSci. But, we are likely looking at anywhere from 4 years or more (likely more) to do both things - and not a small amount of money. After that point, I'll be close to 50 years old. There's a very good and likely chance that the residency won't be around at that point; or even the possibility that technology around ML may have radically changed the world to the point where it would be difficult (or even meaningless) to try to "catch up"... Still - I'm not going to let that possibility stop or restrain me; that said, is it viable for me to even think about this kind of thing - doing such a residency? Would I even have a chance of being considered, vs someone younger? Furthermore - how does one do a residency of this nature, while still paying bills (I live in Phoenix, AZ - I own a house, plus have other bills)? These are just questions I have - not really concerns, as I am no where near the point I need to be to do the residency - and there isn't any guarantee that I will be. I'll just continue to enjoy to journey, rather than worrying about a specific end-goal.
- deleted 10y ago

deepnotderp 10y ago

This is pretty old, and neural nets can train neural nets too (better than humans as usual). Check learning to learn w/ gradient descent by gradient descent

westoncb 10y ago
Err.. this paper cites the one you're referring to, so I do not think it's older (also, the dates they were published).
deepnotderp 10y ago
(Learning to learn is older, I just thought it would be cool)

jorgemf 10y ago

I would love to see a Neural Architecture Search that create Neural Architecture Search as a future research project. Meta-meta-learning. I like the idea of improving the network which creates other networks. Also the size of the network can be used as part of the evaluation in order to minimize the networks and maximize the accuracy.

deleted 10y ago
[deleted]
westoncb 10y ago
I would guess the computational complexity involved in training the top level network would make the project infeasible.
- jorgemf 10y ago
  You can relax the problem in order to reduce complexity and calculations. For example: cut the training of slow learning networks, have a database of already trained networks, decrease the number of epochs and examples, etc. Or even create another network which predicts the convergence of a network and use it as heuristic. If you also take into account the size of the network and force the learning to minimize them, then you can train them faster. We cannot do it at home, but for sure Google can try it in their servers.

Smerity 10y ago

Extreme paper tldr - Humans usually construct neural network components and the graph of how they fit together by hand. This work sets up a "controller" neural network that constructs two core components in many neural networks, an RNN and a CNN, through reinforcement learning. This is an intensive and slow process, requiring 400 CPUs and 800 GPUs for the RNN and CNN respectively, but achieves better than or near state of the art results for language modeling and vision classification respectively. This paper is currently under review for ICLR 2017 and is one of the papers I was most excited about. I previously wrote an article, "In deep learning, architecture engineering is the new feature engineering"[1], which discusses the (often ignored) fact that much of the work in modern deep learning papers is just assembling components in different combinations. This is one of the first works that I feel provides a viable answer to my complaint/concern. The paper itself tackles two problems - first, that optimizing architecture is usually black magic directed poorly by humans, and second, humans rarely spend their time tailoring towards a specific task, instead seeking generality. Zoph and Le do this by having one neural network generate the architecture for a later one through a large series of experiments. They perform experiments in both vision (classification) and text (language modeling), replacing the convolutional neural network component and the recurrent neural network component respectively. First is that many of the choices regarding constructing the neural network architecture are somewhat arbitrary and only hit upon experimentally by the practitioners themselves. Andrej Karpathy noted in one of his lectures (paraphrased) "Start with an architecture that works, then modify from there" - mainly as there's a lot of "black magic" in these architectures that has only been discovered by spilling blood to the experimental god of a hundred GPUs and/or "graduate student descent" (i.e. where you lock a poor grad in a room for an indeterminate period of time and tell them to do better on task X). Being able to turn to a neural network to run this painful search for you instead is a good idea - assuming you have the large number of GPUs or CPUs necessary. In the paper they use 400 CPUs for the language modeling search and 800 GPUs for the CNN classification search! The other is whether we should generalize or specialize these architectures. There are many variants of architectures that are not built for or tested against each possible new task. For example, within recurrent neural networks (RNNs) we have the RNN/GRU/LSTM/QRNN/RHN/... and a million minor minor variants between them, each of which perform slightly different depending on the task. While we would like to imagine the architectures that humans make would get progressively closer to "the perfect generic RNN cell" over time, it makes sense that certain cells are, could, or should be optimized to a specific task. Seeking generality isn't always the correct answer. Humans want to seek generality as we don't have the time to tailor to each specific task - but what if we could? Maybe in that situation Occam's razor is actually an impediment to our thinking. While this is early days, and hugely resource intensive, it is likely to get more feasible over time either as we get more computing power or become smarter regarding how we use it. As a researcher in neural networks, I don't consider this a threat, but instead a useful tool, likely in the same way that a compiler likely only helped assembly programmers. If people are interested, I can write an article covering many of the details of this paper like I did for Google's Neural Machine Translation architecture[2]. In that article I try to step through how these systems work from the ground up, and the reasoning behind many of the decisions in the paper, hopefully in an understandable manner for a general audience. P.S. Merity et al. is one of the numbers they beat in the language modeling section, so you may read this entire post in a bitter tone if you'd like ;) P.P.S. This paper has been out since November 2016 or earlier - I think it was a recent MIT Tech Review article that might have resurfaced it? (oops: wrote Wired initially, meant MIT Tech Review - thanks @saycheese) [1]: http://smerity.com/articles/2016/architectures_are_the_new_feature_engineering.html http://smerity.com/articles/2016/architectures_are_the_new_f... [2]: http://smerity.com/articles/2016/google_nmt_arch.html http://smerity.com/articles/2016/google_nmt_arch.html

jordansmithnz 10y ago

Wait... if the neural net can design other neural nets, can it be taught to design itself?

mastazi 10y ago
Ironically, this is one of the oldest ideas in computer science https://en.wikipedia.org/wiki/Von_Neumann_universal_constructor https://en.wikipedia.org/wiki/Von_Neumann_universal_construc...
- Houshalter 10y ago
  Von Neumanns constructor is about self replication. Not quite an AI that can self improve. I think the first person to put the idea forward was I. J. Good in 1962. He speculated that someday AIs would be good enough to do AI research better than their human masters. Then they would start making even better AIs. Which would make even better AIs, and so on. Leading to what he called an "intelligence explosion". He thought it would be "the last invention that man need ever make." http://web.archive.org/web/20160428183531/http://le-cretin-transnational.ch/wp-content/uploads/2014/04/Good-Speculations-Concerning-the-First-Ultraintelligent-Machine.pdf http://web.archive.org/web/20160428183531/http://le-cretin-t...
  - mastazi 10y ago
    Parent's comment was about self-replication, I was addressing that.
EvgeniyZh 10y ago
Can it be taught to design a better version of itself?