4 ms·
Practical use: Once you have GNNs that predict the trained weights of headless models for vision, NLP, and other domains, you can test new architectures in all
by cs702 5y ago
Practical use: Once you have GNNs that predict the trained weights of headless models for vision, NLP, and other domains, you can test new architectures in all those domains much faster with fewer compute resources.
Research use: Can we figure out how and why these GNNs can predict trained weights for previously unseen architectures without actually having to train the models? Can we simplify the GNNs? Can we use the knowledge gained from studying these GNNs to speed-up conventional training?
- webmaven 5y ago> Practical use: Once you have GNNs that predict the trained weights of headless models for vision, NLP, and other domains, you can test new architectures in all those domains much faster with fewer compute resources. Aha. Using this as an accelerator for Neural Architecture Search would indeed be very neat.
- hwers 5y agoYou still need to train 1 million architectures though before you can start the search (dataset inflexible)… I suppose there’s some value in possibly finding radically different networks (though I wonder if it’d fare well in such outlier regions). Perhaps if/since the GNN model is also differentiable you could invert it to maximize some compute/quality trade off point.
- joshuamorton 5y agoBut like it already exists. That's a sunk cost. Presuming it actually works and gives useful weights, if it lowers training time of novel models by 10% or something, that's dollars saved in the long run.
- deleted 5y ago[deleted]