5 ms·

Darts: Differentiable architecture search for convolutional, recurrent networks

51 points by kumaranvpl 8y ago

JustFinishedBSG 8y ago
I'm pleasantly surprised by how well it works! No offense to Barret Zoph, Quoc Le et al but the "let's throw 10000 GPUs and TPUs at the problem" approach was frankly absurd. And a huge +1 for the inclusion of random architectures in the results.
- osipov 8y ago
  Why do you think that throwing compute at the problem is absurd? Arguably that's the approach that produced the resurgence of interest in CNNs.
  - smaddox 8y ago
    I'm not the original commenter, but I think it's fair to say that throwing compute at the problem is absurd from an academic point of view (where the goal is understanding), but not from a commercial point of view. Although there's still some academic value in throwing compute at a problem, just to see what's possible.
  - chadmeister 8y ago
    Yes I think the whole point of Zoph and Le's research is to empirically demonstrate the desired trade off of compute cycles vs feature engineering
- NhanH 8y ago
  Can you link the paper of Zoph,Le that did the “1000 GPUs” thing? Thanks! I tried searching for it but apparently there was quite a few papers on parameter search.
  - kripke 8y ago
    Not GP, but you are probably looking for the "Neural Architecture Search" series [1] [2] [3]. First one uses something like 1k GPUs for a month, next one is a bit more reasonable, and the last one actually has a reasonable training time, and has comparable performance to DARTS (see ENAS in comparison tables). 1: https://arxiv.org/abs/1611.01578 https://arxiv.org/abs/1611.01578 2: https://arxiv.org/abs/1707.07012 https://arxiv.org/abs/1707.07012 3: https://arxiv.org/abs/1802.03268 https://arxiv.org/abs/1802.03268
ashelmire 8y ago
Cool, was thinking about this subject earlier today. Abstract: >This paper addresses the scalability challenge of architecture search by formulating the task in a differentiable manner. Unlike conventional approaches of applying evolution or reinforcement learning over a discrete and non-differentiable search space, our method is based on the continuous relaxation of the architecture representation, allowing efficient search of the architecture using gradient descent. Extensive experiments on CIFAR-10, ImageNet, Penn Treebank and WikiText-2 show that our algorithm excels in discovering high-performance convolutional architectures for image classification and recurrent architectures for language modeling, while being orders of magnitude faster than state-of-the-art non-differentiable techniques. Zoph's paper on architecture search: https://arxiv.org/abs/1611.01578 https://arxiv.org/abs/1611.01578 Concept of architecture search from ages ago: https://static1.squarespace.com/static/58e2a71bf7e0ab3ba886cea3/t/5909113c1b631b40f8137956/1493766462349/1989+neural+networks.pdf https://static1.squarespace.com/static/58e2a71bf7e0ab3ba886c...
aabajian 8y ago
Not to get existential here, but the analogy to human neural networks is compelling. The algorithm efficiently redesigns itself to fit the data, just like neural synapses reconnect one another as knowledge is learned, to optimize the storage and retrieval of the information.

51 points by kumaranvpl 8y ago

I'm pleasantly surprised by how well it works! No offense to Barret Zoph, Quoc Le et al but the "let's throw 10000 GPUs and TPUs at the problem" approach was frankly absurd. And a huge +1 for the inclusion of random architectures in the results.

osipov 8y ago
Why do you think that throwing compute at the problem is absurd? Arguably that's the approach that produced the resurgence of interest in CNNs.
- smaddox 8y ago
  I'm not the original commenter, but I think it's fair to say that throwing compute at the problem is absurd from an academic point of view (where the goal is understanding), but not from a commercial point of view. Although there's still some academic value in throwing compute at a problem, just to see what's possible.
- chadmeister 8y ago
  Yes I think the whole point of Zoph and Le's research is to empirically demonstrate the desired trade off of compute cycles vs feature engineering
NhanH 8y ago
Can you link the paper of Zoph,Le that did the “1000 GPUs” thing? Thanks! I tried searching for it but apparently there was quite a few papers on parameter search.
- kripke 8y ago
  Not GP, but you are probably looking for the "Neural Architecture Search" series [1] [2] [3]. First one uses something like 1k GPUs for a month, next one is a bit more reasonable, and the last one actually has a reasonable training time, and has comparable performance to DARTS (see ENAS in comparison tables). 1: https://arxiv.org/abs/1611.01578 https://arxiv.org/abs/1611.01578 2: https://arxiv.org/abs/1707.07012 https://arxiv.org/abs/1707.07012 3: https://arxiv.org/abs/1802.03268 https://arxiv.org/abs/1802.03268

ashelmire 8y ago

Cool, was thinking about this subject earlier today. Abstract: >This paper addresses the scalability challenge of architecture search by formulating the task in a differentiable manner. Unlike conventional approaches of applying evolution or reinforcement learning over a discrete and non-differentiable search space, our method is based on the continuous relaxation of the architecture representation, allowing efficient search of the architecture using gradient descent. Extensive experiments on CIFAR-10, ImageNet, Penn Treebank and WikiText-2 show that our algorithm excels in discovering high-performance convolutional architectures for image classification and recurrent architectures for language modeling, while being orders of magnitude faster than state-of-the-art non-differentiable techniques. Zoph's paper on architecture search: https://arxiv.org/abs/1611.01578 https://arxiv.org/abs/1611.01578 Concept of architecture search from ages ago: https://static1.squarespace.com/static/58e2a71bf7e0ab3ba886cea3/t/5909113c1b631b40f8137956/1493766462349/1989+neural+networks.pdf https://static1.squarespace.com/static/58e2a71bf7e0ab3ba886c...

aabajian 8y ago

Not to get existential here, but the analogy to human neural networks is compelling. The algorithm efficiently redesigns itself to fit the data, just like neural synapses reconnect one another as knowledge is learned, to optimize the storage and retrieval of the information.