3 ms·

Can anyone enlighten me how you'd make use of this practically though? It seems like you need like a dataset of a million models to train this beast (all of whi

by hwers 5y ago

Can anyone enlighten me how you'd make use of this practically though? It seems like you need like a dataset of a million models to train this beast (all of which trained on a specific dataset). Yeah it can figure out the weights for a new model but you had to train 1 million models first to be able to even do that. Am I missing something?

cs702 5y ago
Practical use: Once you have GNNs that predict the trained weights of headless models for vision, NLP, and other domains, you can test new architectures in all those domains much faster with fewer compute resources. Research use: Can we figure out how and why these GNNs can predict trained weights for previously unseen architectures without actually having to train the models? Can we simplify the GNNs? Can we use the knowledge gained from studying these GNNs to speed-up conventional training?
- webmaven 5y ago
  > Practical use: Once you have GNNs that predict the trained weights of headless models for vision, NLP, and other domains, you can test new architectures in all those domains much faster with fewer compute resources. Aha. Using this as an accelerator for Neural Architecture Search would indeed be very neat.
  - hwers 5y ago
    You still need to train 1 million architectures though before you can start the search (dataset inflexible)… I suppose there’s some value in possibly finding radically different networks (though I wonder if it’d fare well in such outlier regions). Perhaps if/since the GNN model is also differentiable you could invert it to maximize some compute/quality trade off point.
    - joshuamorton 5y ago
      But like it already exists. That's a sunk cost. Presuming it actually works and gives useful weights, if it lowers training time of novel models by 10% or something, that's dollars saved in the long run.
      - deleted 5y ago
        [deleted]
nestorD 5y ago
If you work on a particular problem (lets say because it is your startup's object of research) you very likely are testing a large number of architectures on your dataset. With this you could train a GNN on the various models you tested to speed-up training new ones.

by hwers 5y ago

cs702 5y ago

Practical use: Once you have GNNs that predict the trained weights of headless models for vision, NLP, and other domains, you can test new architectures in all those domains much faster with fewer compute resources. Research use: Can we figure out how and why these GNNs can predict trained weights for previously unseen architectures without actually having to train the models? Can we simplify the GNNs? Can we use the knowledge gained from studying these GNNs to speed-up conventional training?

webmaven 5y ago
> Practical use: Once you have GNNs that predict the trained weights of headless models for vision, NLP, and other domains, you can test new architectures in all those domains much faster with fewer compute resources. Aha. Using this as an accelerator for Neural Architecture Search would indeed be very neat.
- hwers 5y ago
  You still need to train 1 million architectures though before you can start the search (dataset inflexible)… I suppose there’s some value in possibly finding radically different networks (though I wonder if it’d fare well in such outlier regions). Perhaps if/since the GNN model is also differentiable you could invert it to maximize some compute/quality trade off point.
  - joshuamorton 5y ago
    But like it already exists. That's a sunk cost. Presuming it actually works and gives useful weights, if it lowers training time of novel models by 10% or something, that's dollars saved in the long run.
    - deleted 5y ago
      [deleted]

nestorD 5y ago

If you work on a particular problem (lets say because it is your startup's object of research) you very likely are testing a large number of architectures on your dataset. With this you could train a GNN on the various models you tested to speed-up training new ones.