The simplest approach (training NNs from scratch, using validation score as loss, simple DRL agent with REINFORCE as the designer) requires thousands of models

The simplest approach (training NNs from scratch, using validation score as loss, simple DRL agent with REINFORCE as the designer) requires thousands of models to be trained and exorbitant GPU resources, like the Zoph paper. You can, however, get human-level designs with a few dozen or hundred samples at a tiny fraction of the computation cost if you are somewhat smarter about it - for example, reusing the parameters of a trained NN when you start training a new slightly-different NN (eg if you have a CNN with 20 layers and you want to try the exact same settings but with 21 layers, almost all of that CNN is going to be very similar to the end result of the 20 layers, so you can speed things up by copying over the first 20 layers, randomly initializing the 21st layer, and then training it for a short time). One nice efficient form of neural architecture search is SMASH: https://www.reddit.com/r/reinforcementlearning/comments/6uioms/smash_oneshot_model_architecture_search_through/ https://www.reddit.com/r/reinforcementlearning/comments/6uio... More links on the general topic of 'DL optimizing/designing DL': https://www.reddit.com/r/reinforcementlearning/search?q=flair%3ADL+flair%3AMetaRL&restrict_sr=on https://www.reddit.com/r/reinforcementlearning/search?q=flai... (It's pretty critical to current DL approaches to few-shot/one-shot learning as well: you can think of it as 'designing' a new NN specialized to the few samples of the new class of data.)