5 ms·

Really naive question: How many deep learning models would need to be trained so that a model (maybe a decision tree) for generating deep learning models could

by afpx 8y ago

Really naive question: How many deep learning models would need to be trained so that a model (maybe a decision tree) for generating deep learning models could be trained?

RoboTeddy 8y ago
This question isn't silly! People have actually trained neural networks to generate other neural networks. The "designer" network is trained using reinforcement learning, with a reward signal that depends on the efficacy of the networks it creates. See: Neural Architecture Search with Reinforcement Learning (https://arxiv.org/abs/1611.01578 https://arxiv.org/abs/1611.01578) It looks like they trained the "designer" network by trainning 15,000 deep learning models (800 at a time in parallel over 800 GPUs)
gwern 8y ago
The simplest approach (training NNs from scratch, using validation score as loss, simple DRL agent with REINFORCE as the designer) requires thousands of models to be trained and exorbitant GPU resources, like the Zoph paper. You can, however, get human-level designs with a few dozen or hundred samples at a tiny fraction of the computation cost if you are somewhat smarter about it - for example, reusing the parameters of a trained NN when you start training a new slightly-different NN (eg if you have a CNN with 20 layers and you want to try the exact same settings but with 21 layers, almost all of that CNN is going to be very similar to the end result of the 20 layers, so you can speed things up by copying over the first 20 layers, randomly initializing the 21st layer, and then training it for a short time). One nice efficient form of neural architecture search is SMASH: https://www.reddit.com/r/reinforcementlearning/comments/6uioms/smash_oneshot_model_architecture_search_through/ https://www.reddit.com/r/reinforcementlearning/comments/6uio... More links on the general topic of 'DL optimizing/designing DL': https://www.reddit.com/r/reinforcementlearning/search?q=flair%3ADL+flair%3AMetaRL&restrict_sr=on https://www.reddit.com/r/reinforcementlearning/search?q=flai... (It's pretty critical to current DL approaches to few-shot/one-shot learning as well: you can think of it as 'designing' a new NN specialized to the few samples of the new class of data.)

by afpx 8y ago

Really naive question: How many deep learning models would need to be trained so that a model (maybe a decision tree) for generating deep learning models could be trained?

RoboTeddy 8y ago

This question isn't silly! People have actually trained neural networks to generate other neural networks. The "designer" network is trained using reinforcement learning, with a reward signal that depends on the efficacy of the networks it creates. See: Neural Architecture Search with Reinforcement Learning (https://arxiv.org/abs/1611.01578 https://arxiv.org/abs/1611.01578) It looks like they trained the "designer" network by trainning 15,000 deep learning models (800 at a time in parallel over 800 GPUs)

gwern 8y ago

The simplest approach (training NNs from scratch, using validation score as loss, simple DRL agent with REINFORCE as the designer) requires thousands of models to be trained and exorbitant GPU resources, like the Zoph paper. You can, however, get human-level designs with a few dozen or hundred samples at a tiny fraction of the computation cost if you are somewhat smarter about it - for example, reusing the parameters of a trained NN when you start training a new slightly-different NN (eg if you have a CNN with 20 layers and you want to try the exact same settings but with 21 layers, almost all of that CNN is going to be very similar to the end result of the 20 layers, so you can speed things up by copying over the first 20 layers, randomly initializing the 21st layer, and then training it for a short time). One nice efficient form of neural architecture search is SMASH: https://www.reddit.com/r/reinforcementlearning/comments/6uioms/smash_oneshot_model_architecture_search_through/ https://www.reddit.com/r/reinforcementlearning/comments/6uio... More links on the general topic of 'DL optimizing/designing DL': https://www.reddit.com/r/reinforcementlearning/search?q=flair%3ADL+flair%3AMetaRL&restrict_sr=on https://www.reddit.com/r/reinforcementlearning/search?q=flai... (It's pretty critical to current DL approaches to few-shot/one-shot learning as well: you can think of it as 'designing' a new NN specialized to the few samples of the new class of data.)